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Introduction 


Although  complementary  mental  models  may  be  a  distinction  novel  to  this  thesis,  the 
proposal  that  shared  mental  models  exist  is  not.  Extensive  research  has  been  conducted  in  this 
area.  There  also  has  been  significant  related  work  outlining  cognitive  constructs  at  the  team  level 
such  as  distributed  cognition  (Hutchins,  1991),  team  decision  making,  shared  mental  models,  and 
team  situation  awareness  (Cannon-Bowers,  Salas,  and  Converse,  1993;  Cooke  et  al.,  in  press; 
Orasanu,  1990;  Stout,  Cannon-Bowers,  and  Salas,  1996).  While  this  literature  review  focuses  on 
the  development  of  team  mental  models,  some  of  the  concepts  discussed  overlap  with  the 
theoretical  constructs  listed  above. 


Military  significance 

Crew  coordination  and  team  performance  are  extremely  important  to  military  small  teams 
and  their  missions.  Obtaining  measurements  of  these  constructs  within  participant  samples  is 
essential  for  better  understanding  of  the  components  of  both  crew  coordination  and  team 
performance,  yet  to  date  measures  have  mostly  been  of  a  subjective  nature.  Enhanced  study  of 
the  roles  of  team  members  during  team  tasks  and  emergency  procedures  will  add  greatly  to  our 
knowledge  about  team  performance  and  its  relationship  to  information-sharing  strategies. 

Background  and  literature  review 
What  are  teams? 

“Teams  are  more  than  just  a  gathering  of  persons  pursuing  their  own  goals,  and  teamwork 
is  more  than  just  the  aggregate  of  individual  team  member’s  behaviors”  (Paris,  Salas,  and 
Cannon-Bowers,  2000). 

Even  a  cursory  review  of  the  literature  available  on  teams  reveals  that  no  single  definition  of 
a  team  has  been  universally  accepted  (LaJoie  and  Sterling,  1999).  Then  what  is  a  team?  Salas 
gives  a  generally  accepted  definition  of  a  team  as  a  collection  of  (two  or  more)  individuals 
working  together  interdependently  to  achieve  a  common  goal  (Salas  et  al.,  1992).  For  the 
purpose  of  clarity,  in  this  research  a  team  is  further  defined  as  a  small  number  of  cognitive 
agents,  each  with  defined  roles  and  responsibilities,  task  relevant  knowledge  and  common  goals. 
Their  task  requires  more  than  one  information  source  and  interdependence  and  coordination 
among  members  (Orasanu  and  Salas,  1993;  Paris,  Salas,  and  Cannon-Bowers,  2000).  Although 
this  research  effort  focuses  on  the  interactions  between  two  team  members  in  close  proximity 
conducting  a  task  that  requires  continuous  coordination,  it  is  important  to  note  that  the  structure 
of  a  team  may  vary  from  highly  structured  interdependent  teams  to  teams  whose  members 
interact  little  and  perform  tasks  in  the  same  location  (Salas  et  al.,  1992). 

One  of  the  most  important  characteristics  of  a  team  that  is  identified  in  nearly  all  definitions 
is  the  existence  of  common  goals.  Common  goals  are  crucial  to  successful  teamwork;  they  help 
bind  a  team  together,  instill  a  vested  interest  in  each  other’s  performance,  and  reduce  self- 
centered  actions.  Furthermore,  realizing  team  goals  contributes  to  a  team’s  mutual  awareness  of 
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not  only  the  goals,  but  also  other  information  requirements  such  as  team  member  roles,  team 
structure  and  task  structure  (what  information  is  required  for  each  task).  Along  with  other 
characteristics  of  “good”  team  performance,  teams  must  maintain  an  on-going  dialogue  to 
continuously  exchange  these  information  requirements.  This  mutual  awareness  is  described  as  a 
team  “shared  mental  model”  (Rouse,  Cannon-Bowers,  and  Salas,  1992;  Cannon-Bowers,  Salas, 
and  Converse,  1993).  Consequently,  the  development  of  shared  mental  models  among  team 
members  is  a  common  goal  of  training  programs  in  order  to  increase  performance  (Volpe  et  ah, 
1996;  Cannon-Bowers  et  ah,  1998),  but  has  not  been  a  focus  of  system  design.  For  example, 
Volpe  et  ah  (1996)  looked  at  the  effect  of  cross  training  on  team  process  and  performance.  They 
found  that  cross  training  improved  team  member’s  inter-positional  knowledge  and  team 
performance.  This  increase  in  performance  was  theorized  to  be  related  to  an  improved  team 
shared  mental  model  (Volpe  et  ah,  1996). 

Small  groups  are  also  considered  in  this  review.  Groups  differ  from  teams  in  that  they  may 
not  have  specific  roles  and  can  be  loosely  assembled  without  a  common  goal  or  shared  history 
(McIntyre  and  Salas,  1995).  Klimoski  and  Mohammed  identified  another  distinction  between 
teams  and  groups:  “groups  are  collections  of  individuals  whose  tenure  together  and  division  or 
responsibilities  can  vary  considerably  ...a  team  consists  of  differentiated  and  interdependent 
members.  All  teams  are  groups,  but  the  converse  is  not  necessarily  so.”  (Klimoski  and 
Mohammed,  1994).  There  is  a  gray  area  between  groups  and  teams  in  which  the  military 
commonly  operates.  During  many  military  operations,  and  likewise,  a  condition  present  during 
experimentation  in  this  thesis,  although  roles  are  specified  and  the  crew  shares  common  goals, 
crew  composition  may  be  improvised  for  a  specific  mission;  this  ad  hoc  collection  of  people  may 
sometimes  operate  as  a  team  and  sometimes  as  a  small  group. 

Mental  models 

An  aspect  of  groups  and  teams  that  directly  affects  group  processes  is  the  compatibility  of 
teams’  mental  models.  Therefore,  it  is  important  to  discuss  what  mental  models  are,  how  they 
are  developed,  and  finally  what  is  shared  in  a  shared  mental  model  that  individuals  hold  about 
their  team,  task,  etc.  This  section  focuses  on  the  topic  of  mental  models  in  terms  of  individual 
members  and  shared  representations  within  a  team. 

“Mental  models  are  the  mechanisms  whereby  humans  are  able  to  generate  descriptions  of 
system  purpose  and  form  explanations  of  system  functioning  and  observed  system  states,  and 
predictions  (or  expectations)  of  future  system  states”  (Rouse  and  Morris,  1986).  Essentially, 
mental  models  are  organized  knowledge  structures  that  allow  individuals  to  interact  with  their 
environment.  A  mental  model  is  a  type  of  knowledge  representation  or  structure  that  individuals 
possess  concerning  their  interaction  with  the  world  (Norman,  1983).  Specifically,  mental  models 
allow  people  to  describe,  explain  and  predict  the  behavior  of  the  world  around  them  (see  Figure 
1),  to  recognize  and  remember  relationships  among  components  of  the  environment,  and  to 
construct  expectations  for  what  is  likely  to  occur  next  (Orasanu,  1993;  Rouse,  Cannon-Bowers, 
and  Salas,  1992;  Rouse  and  Morris,  1986).  This  implies  that  a  specific  mental  model  of  a  system 
state  may  be  sufficient  or  even  advanced  for  one  function  (e.g.,  describing)  yet  inadequate  for 
another  (e.g.,  predicting).  The  mental  model  construct  also  has  been  used  as  a  means  to  evaluate 
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the  nature  of  an  operator’s  knowledge  of  complex  system  performance,  and  as  a  basis  to  analyze 
effective  and  ineffective  performance.  Gentner  and  Stevens  (1983)  and  Rumelhart  and  Ortony 
(as  cited  in  Smith-Jentsch  et  al.,  1998)  indicated  that  mental  models  also  guide  team  members’ 
interactions  with  others. 


Purpose 


Function 


State 


Form 


>  Why  a  system  exists 


*•  How  a  system  operates 


+■  What  a  system  is  doing 


What  a  system  looks  like 


Figure  1.  Nature  of  mental  models  (Rouse,  Cannon-Bowers, 
and  Salas.  1992). 


Research  cited  in  the  previous  paragraph  suggests  that  different  individuals  may  possess 
different  yet  accurate  mental  models.  Accurate  mental  models  give  an  individual  the  ability  to 
retrieve  information  relevant  to  a  task  quicker  than  if  they  possessed  an  inaccurate  model.  More 
simply,  individuals  with  an  accurate  mental  model  will  be  more  likely  to  correctly  interpret  and 
respond  to  a  situation  that  confronts  them  (Stout,  Salas,  and  Fowlkes,  1997).  Mental  models  may 
contain  various  types  and  levels  of  information.  For  instance,  Jonassen  and  Tessmer  (1996)  say 
that  mental  models  contain  information  concerning  declarative  knowledge,  conditional 
knowledge,  procedural  skills,  and  functions.  Rouse,  Cannon-Bowers,  and  Salas  (1992)  comment 
on  the  dimension  of  knowledge.  They  assert  that  the  level  of  knowledge  within  the  mental  model 
construct  can  range  from  detailed,  specific,  or  concrete  to  global,  general,  or  abstract  (Rouse, 
Cannon-Bowers,  and  Salas,  1992). 

The  knowledge  structure  of  individual  mental  models  includes  an  equipment  model,  task 
model,  team  interaction  model,  and  team  model  (Rouse,  Cannon-Bowers,  and  Salas,  1992; 
Cannon-Bowers,  Salas,  and  Converse,  1993): 
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•  Equipment  knowledge  relates  to  equipment  functioning,  limitations,  operating 
procedures,  and  likely  failures. 

•  Task  knowledge  considers  task  procedures,  strategies,  contingencies  and  scenarios,  and 
environmental  constraints. 

•  Team  knowledge  includes  the  following  dimensions  of  teams:  (1)  a  team  interaction 
model  incorporating  roles  and  responsibilities,  sources  of  information,  patterns  of 
interaction  and  communication,  and  interdependencies  among  members;  and  (2)  a  team 
model  containing  the  knowledge,  skills,  abilities,  and  behavioral  tendencies  of  team 
members. 

Part  of  team  knowledge  consists  of  what  each  team  member  understands  about  teamwork. 
Although  team  members  may  have  teamwork  knowledge  within  a  specific  domain,  (e.g., 
teamwork  on  a  football  team  or  an  aircraft  cockpit),  it  is  likely  that  they  will  also  acquire  and 
develop  a  core  teamwork  knowledge  structure  that  applies  to  most  types  of  teams  (Gibson  and 
Zellmer,  1998).  This  might  include  communications,  relationships,  and  so  on  (Salas  et  ah,  1988). 
Core  teamwork  knowledge  provides  individuals  with  an  understanding  and  a  structure  for 
working  together  as  a  team. 

These  four  types  of  knowledge  that  form  mental  models  can  be  viewed  as  reflecting  two 
major  content  domains:  (1)  task-related  features  of  the  situation  (e.g.,  the  technology/equipment 
and  job/task  models),  and  (2)  team-related  aspects  of  the  situation  (e.g.,  the  team  interaction  and 
team  models).  This  division  is  consistent  with  the  idea  that  teams  develop  two  tracks  of  behavior: 
a  teamwork  track  and  a  task-work  track  (Morgan  et  ah,  1986;  McIntyre  and  Salas,  1995;  Mathieu 
et  ah,  2000). 

The  study  of  mental  models  places  emphasis  on  the  organization  of  information  as  it  is 
stored  in  memory,  not  necessarily  the  amount  of  information  (Johnson-Laird,  1983;  Rouse  and 
Morris,  1986).  This  organization  of  information  as  stored  in  an  individual’s  memory  is  of  great 
import  to  the  study  of  teamwork.  For  example,  Wegner  (1987)  described  the  idea  of  transactive 
memory  systems,  analogizing  team  information  sharing  as  an  expansion  of  personal  memory 
through  interdependence  on  team  members.  He  explained  that,  just  like  all  parts  of  an 
individual's  recall  system  do  not  contain  the  same  knowledge;  neither  does  each  member  of  a 
team.  Yet,  effective  teams  are  aware  of  who  possesses  certain  information  (Wegner,  1987). 
Therefore,  if  team  member's  access  to  task  critical  information  is  focused  on  their  roles  and  tasks, 
labels  identifying  "who  knows  what  information"  are  naturally  created. 

Shared  mental  models 

This  study  focuses  on  the  processes  involved  in  the  acquisition,  storage,  transmission,  and 
use  of  information  for  the  purposes  of  improving  team  performance.  Although  acquisition, 
storage,  transmission  and  use  of  information  occurs  independent  of  team  affiliation  to  form  an 
individual’s  mental  model,  the  process  is  consistent  with  views  in  small  group  and  team  literature 
that  consider  intra-personal  communication  as  a  form  of  information  processing  (Gibson  and 
Zellmer,  1998).  This  processing  has  been  referred  to  as  transactive  memory,  as  described  above 
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(Wegner,  1987),  team  or  shared  mental  models  (Rouse,  Cannon-Bowers,  and  Salas,  1992; 
Klimoski  and  Mohammed,  1994),  distributed  cognition  (Hutchins  and  Klausen  1996),  collective 
cognition  (Gibson,  1996),  and  teamwork  knowledge  schemas  (Rentsch,  Heffner,  and  Duffy, 

1994;  Gibson  and  Zellmer,  1998).  Upon  examination,  each  of  these  terms  fundamentally  refers 
to  the  level  of  congruence  of  individual  knowledge  structures  between  members  of  a  small  group 
or  team.  This  review  will  use  the  term  “shared  mental  model”  as  the  primary  reference  to  this 
relationship  between  knowledge  structures  of  team  members. 

Klimoske  and  Mohammed  (1994)  contended  “there  can  be  (and  probably  would  be)  multiple 
mental  models  co-existing  among  team  members  at  a  given  point  in  time.”  As  discussed  in  the 
previous  section,  these  would  include  models  of  task  (e.g.,  navigation),  equipment  (e.g., 
technology),  teamwork,  and  interactions  (e.g.,  communications).  Teams  with  members  that  share 
similar  knowledge  structures  regarding  the  task,  the  environment,  equipment,  member 
capabilities,  and  member  interactions  communicate  more  effectively  and  perform  better  than 
teams  whose  members  do  not  share  such  knowledge  (Klimoske  and  Mohammed  1994). 
According  to  Cannon-Bowers,  Salas,  and  Converse  (1993),  complex  tasks  probably  require  that 
multiple  mental  models  be  shared  among  members.  The  concept  of  closely  related  mental 
models  among  team  members  has  been  offered  as  a  means  to  explain  coordinated  performance  in 
teams,  especially  in  conditions  of  high  workload  (Cannon-Bowers,  Salas,  and  Converse,  1993; 
Rouse,  Cannon-Bowers,  and  Salas,  1992).  The  benefits  of  shared  mental  models  are 
unmistakably  evident  when  a  team  is  conducting  a  complex  task  and/or  operating  in  a  complex 
environment.  Shared  mental  models  help  team  members  form  accurate  expectations  of  the  task 
and  each  other  (Cannon-Bowers,  Salas,  and  Converse,  1993;  Espinosa  et  al.,  2001;  Klimoski  and 
Mohamed,  1994;  Rouse  and  Morris,  1986).  Recent  studies  show  that  team  performance  at  tasks 
requiring  anticipation  of  team  member’s  actions  and  information  requirements  is  improved  with 
enhanced  shared  mental  models  (Espinosa  et  al.,  2001). 

The  notion  of  shared  mental  models  in  teams  has  received  a  considerable  amount  of 
attention  in  the  literature  in  recent  years.  Shared  mental  models  are  thought  to  provide  team 
members  with  a  common  understanding  of  who  is  responsible  for  what  task  and  what  the 
information  requirements  are  and  to  allow  team  members  to  anticipate  one  another’s  needs  so 
that  they  can  work  in  sync  and  adjust  their  behavior  accordingly  (Smith-Jentsch,  Johnston,  and 
Payne,  1998).  The  shared-mental-model  construct  suggests  that  it  is  not  only  the  overlap  of 
knowledge  among  team  members  that  is  predictive  of  team  outcomes  but  also  the  synergy  of  the 
knowledge  organizations  (Mathieu  et  al.,  2000). 

Forming  a  shared  mental  model  of  the  team,  the  task,  and  the  informational  requirements  of 
team  members,  serves  as  an  important  mechanism  for  achieving  efficient  communications  and 
overall  improved  team  performance  (Stout  et  al.,  1999).  When  team  members  share  accurate 
mental  models  of  the  teamwork  processes  that  influence  their  performance,  they  should  be  better 
able  to  uncover  performance  trends  and  diagnose  deficiencies,  focus  their  practice  appropriately 
on  specific  goals,  and  generalize  the  lessons  they  learn  to  new  situations.  The  link  between 
shared  mental  models  and  team  performance  is  evident  in  the  literature.  Numerous  studies  in  the 
past  decade  have  recently  highlighted  the  positive  relationship  between  congruent  shared  mental 
models  and  successful  team  performance  (e.g.,  Smith-Jentsch,  Johnston,  and  Payne,  1998; 
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Blickensderfer,  Cannon-Bowers,  and  Salas,  1994;  Heffner,  Mathieu,  and  Goodwin,  1995; 
Cannon-Bowers  et  al.,  1995;  Minionis,  1994;  Volpe  et  al.,  1996;  Duffy,  1992;  Converse, 
Cannon-Bowers,  and  Salas,  1991). 

Shared  mental  models  and  communication 


“Put  simply,  if  team  members  know  what  to  expect  and  can  explain  what  they  observe,  team 
performance  is  likely  to  be  enhanced”  (Rouse,  Cannon-Bowers,  and  Salas,  1992). 

The  importance  of  communication  to  team  performance  and  the  development  of 
shared  mental  models  cannot  be  overstated.  Communication  is  more  than  an  exchange  of 
information,  it  is  a  means  by  which  teams  coordinate  resources  and  activities  (Entin  and  Serfaty, 
1999),  construct  and  maintain  shared  mental  models  (Orasanu,  1990),  and  establish  and  maintain 
situational  awareness  (Prince  and  Salas,  1993).  There  are  three  basic  types  of  communication 
between  team  members;  (1)  verbal  communication  between  crewmembers;  (2)  non-verbal 
communication  between  crewmembers  (this  could  be  hand  signals,  facial  expressions,  head 
movements  etc.);  and  (3)  written  communication  (this  can  occur  on  kneeboard  spot  reports, 
graphics  on  maps  etc.)  (Orlady  and  Orlady,  2002). 

These  types  of  communication  are  rarely  mutually  exclusive  and  commonly  used  together 
during  operations  in  a  complex  environment  and,  when  necessary,  crews  will  find  ways  to 
communicate.  For  example,  in  an  instance  of  “lost  communications”  between  the  front  and  back 
seat  pilot  of  a  tandem  seated  helicopter  during  a  mission  in  Operation  Desert  Storm”  (all  radios 
were  inoperative  along  with  the  internal  communication  system  in  an  AH-1  Cobra  attack 
helicopter),  the  gunner  was  reduced  to  writing  messages  with  a  grease  pen  on  the  canopy  to  the 
back  seat  pilot;  to  get  the  attention  of  the  gunner  the  back  seat  pilot  used  a  pointer  to  hit  him  on 
the  head. 

Explicit  communication  is  easily  observed.  Through  the  use  of  voice  data  recorders,  an 
analyst  can  review  the  dialogue  of  a  crew  while  watching  a  computer-generated  video  of  the 
aircraft  and  its  critical  flight  data.  Similarly,  pilots  can  be  video  taped  and  audio  taped  in  the 
cockpit  of  a  simulator.  There  are  various  methods  available  to  capture  explicit  communication. 
The  difficulty  arises  in  determining  a  taxonomy  to  determine  what  is  “good  or  bad” 
communication  and  what  is  “improved  or  degraded”  communication.  Entin  and  Entin  (2001) 
have  had  success  in  accurately  capturing  what  communications  occur  among  team  members; 
through  the  use  of  a  verbal  communication  matrix,  they  have  developed  a  taxonomy  by  which 
improvement  in  communication  might  be  measured  in  a  crew  environment  (Figure  2). 
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Measure 

Description 

Overall  Rate 

Total  Communications 

Total  number  of  communications  per  minute 

(Communication  Types 

Information  Requests 
Information  Transfers 
Action  Requests 

Action  T ransfers 

Coordination  Requests 

Coordination  Transfers 

Acknowledgements 

Number  of  requests  for  information  per  minute 
Number  of  Information  Transfers  per  minute 
Number  of  Action  Requests  per  minute 

Number  of  Action  Transfers  per  minute 

Number  of  Coordination  Requests  per  minute 

Number  of  Coordination  Transfers  per  minute 
Number  of  non-substantive  acknowledgements 
of  receipt  of  communication  per  minute  (e.g. 
"OK") 

Communication  Ratios 

Overall  anticipation 

Information  anticipation 
Action  anticipation 

All  communication  transfers  divided  by  all 
communication  requests 

Information  transfers  divided  by  information 
requests 

Action  transfers  divided  by  action  requests 

Figure  2.  Communication  taxonomy  (Entin  and  Entin,  2001). 


The  level  of  team  knowledge  required  to  have  shared  mental  models  can  be  gained  through 
familiarity  with  team  members  and  the  task  at  hand  (Endsley,  1995;  Wegner,  1987),  but  many 
situations  typical  to  operations  in  a  complex  environment  dictate  that  the  team  members  and/or 
the  task  be  novel  and  dynamic.  This  is  the  case  with  many  civilian  and  military  aircraft  crews; 
they  may  not  have  personal  knowledge  of  team  members.  Consequently,  this  team  knowledge 
must  be  acquired  through  other  means.  Teams  that  operate  in  complex  environments,  such  as 
military  teams  and  emergency  teams  in  high  hazard  industries,  need  shared  mental  models,  which 
are  based  on  team  knowledge,  if  they  are  to  operate  successfully.  The  most  effective  teams  seem 
to  share  their  mental  picture  (or  model)  of  the  situation  with  other  team  members  (Stout  et  al., 
1999).  These  shared  mental  models  help  team  members  to  anticipate  the  needs  of  others  and  this 
permits  them  to  either  provide  assistance,  as  it  is  required,  or  to  predict  and  pre-empt  the  need  for 
assistance  (Martin  and  Flin,  1997). 

The  communication  literature  (e.g.,  Johnston  and  Briggs,  as  cited  in  Stout  and  Salas,  1993) 
suggests  interesting  implications  for  the  development  of  shared  mental  models  in  the  use  of 
efficient  communication  strategies.  Providing  information  in  advance  appears  to  be  particularly 
beneficial  in  situations  characterized  by  increased  workload.  Considering  that  Johnston  and 
Briggs  (1968)  theorized  that  communications  are  restricted  in  high-workload  conditions,  it 
appears  that,  in  such  cases,  effective  teams  contain  at  least  one  member  who  continues  to  provide 
information  so  that  others  do  not  need  to  explicitly  request  it  (as  cited  in  Stout  and  Salas,  1993). 

In  routine  situations  with  highly  defined  roles  and  tasks,  shared  mental  models  enable  a  team  to 
function  efficiently  with  little  or  no  explicit  communication;  this  might  occur  in  a  performing  arts 
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crew  or  a  baseball  team  (Orasanu,  1993).  Adaptive  team  communication  and  coordination  skills 
have  their  primary  impact  on  the  team’s  mutual  mental  models  (Serfaty,  Entin,  and  Johnston, 
1998),  and  their  effect  is  enhanced  when  a  team  is  under  stress  and  they  are  more  likely  to  fail  to 
consider  critical  information  even  when  it  is  available  to  them  (Orasanu,  1993). 

Numerous  studies  support  the  concept  that  shared  mental  models  may  lead  to  more  efficient 
and  effective  communication  strategies  (Cannon-Bowers,  Salas,  and  Converse,  1993;  Orasanu, 
1990;  Rouse,  Cannon-Bowers,  and  Salas,  1992).  The  converse  is  also  true,  i.e.,  communication 
is  important  for  the  development  and  maintenance  of  shared  mental  models  (Blickensderfer, 
Cannon-Bowers,  and  Salas,  1997;  Gibson  and  Zellmer,  1998;  Orasanu,  1990;  Rentsch  and  Hall, 
1994;  Stout,  Cannon-Bowers,  and  Salas,  1996).  During  explicit  communication  between  team 
members,  social  cognition  occurs  and  mental  models  develop.  In  the  past,  the  term  social 
cognition  has  referred  to  the  content  of  individual  cognitions  regarding  social  behavior  in 
interaction  with  other  people.  More  recently,  researchers  are  looking  at  a  different  type  of  social 
cognition  in  which  the  word  "social"  denotes  how  cognition  is  accomplished  (Gibson  and 
Zellmer,  1998).  These  explicit  communications  and  interaction  among  team  members  serve  a 
number  of  purposes:  (1)  helping  bring  problem-relevant  information  to  light;  (2)  serving  as  a 
means  of  influencing  the  individual-level  cognitive  processes  that  take  place  within  each  group 
member  (for  example,  by  highlighting  certain  items  of  information  group  members  can  affect  one 
another’s  perceptions,  judgments,  and  opinions  [Stasser  and  Davis,  1981]);  and  (3)  serving  as  the 
vehicle  by  which  group  members’  perceptions,  judgments  and  opinions  are  combined  to  arrive  at 
a  single  solution  to  a  given  problem  (Gibson  and  Zellmer,  1998). 

Three  basic  functions  of  communication  are  to  share  information,  direct  actions  and  reflect 
thoughts  (Orasanu,  1993).  Through  these  functions,  communication  fulfils  two  important 
purposes  concerning  the  development  and  maintenance  of  shared  mental  models.  First,  during 
task  execution,  communication  improves  team  mental  models  with  contextual  cues,  which  may 
result  in  more  precise  predictions  of  the  team  task  (Stout,  Cannon-Bowers,  and  Salas,  1996). 
Second,  communication  ensures  that  the  team  mental  models  are  kept  current  with  regard  to 
changes  that  occur  during  task  execution  (Schraagen  and  Rasker,  2001).  Team  members  must 
communicate  in  dynamic  or  novel  situations;  this  aids  team  members  in  determining  why  a 
previous  strategy  may  not  have  worked,  developing  new  strategies,  reacting  to  and  anticipating 
environmental  cues  and  predicting  future  states  of  the  system  (Orasanu,  1990;  Schraagen  and 
Rasker,  2001).  In  short,  “explicit  and  efficient  communication  assures  that  all  crew  members 
share  an  understanding  of  the  problem,  strategies  for  coping  with  it,  and  who  will  do  what  tasks” 
(Orasanu,  1993). 

Complementary  mental  models 

Throughout  the  review  of  current  literature  concerning  mental  models  presented  above,  it  is 
clear  that,  although  the  expression  “shared  mental  model”  is  commonly  used  throughout  the 
literature,  the  term  “shared”  has  multiple  senses  (Klimoski  and  Mohammed,  1994).  To  share  can 
mean  “to  have  in  common”  (as  in  share  the  equipment  or  share  the  belief),  or  it  can  mean,  “to 
divide”  (e.g.,  share  the  workload)  (Cooke  et  al.,  2000).  Likewise,  shared  information  can  refer  to 
either  information  that  is  homogeneous  with  respect  to  team  members,  or  information  that  is 
distributed  among  team  members  (i.e.,  heterogeneous)  (Cooke  et  al.,  2000).  Because  of  this 
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confusion,  Cooke  et  al.  (2000)  have  refrained  from  using  the  term  “shared”  when  they  discuss 
team  awareness  and  knowledge.  “Team  knowledge”  is  their  preferred  term  because  it  does  not 
imply  the  aspect  of  sharing  that  refers  to  holding  in  common.  Within  team  knowledge  are  team 
mental  models  (a  collective  knowledge  base  of  task  and  team-relevant  information)  and 
situational  models  which  make  use  of  team  mental  models,  but  also  include  situational 
characteristics  and  develop  while  engaged  in  the  task  (Hopp,  Smith,  and  Hayne,  2002). 

Previous  empirical  research  does  not  support  the  view  that  effective  teams  have  shared 
knowledge  or  shared  mental  models  in  the  sense  of  common  or  identical  knowledge.  Instead, 
team  members  might  hold  compatible  or  complementary  knowledge  in  addition  to  common 
knowledge  (Cannon-Bowers  and  Salas,  1997;  Klimoski  and  Mohammed,  1994).  That  is,  there 
might  be  some  knowledge  overlap  required  among  team  members,  but,  in  addition,  role-specific 
yet  compatible  knowledge  is  required  (Cooke  et  al,  2000).  Cooke  explains: 

A  surgical  team  is  an  example  of  knowledge  heterogeneity.  In  some  instances,  the  nurse  and 
the  surgeon  might  need  to  have  some  knowledge  that  is  held  in  common.  However,  the  nurse 
probably  won't  be  able  to  understand,  and  probably  won't  need  to  understand,  all  of  the 
surgeon's  knowledge.  Hence  the  nurse  must  have  some  knowledge  that  is  compatible  with, 
but  not  necessarily  identical  to,  the  surgeon's  knowledge  (Cooke  et  al.,  2000). 

Accordingly,  a  certain  degree  of  overlap  in  team  mental  models  is  preferred,  whereas 
"completely  overlapping  team  mental  models  are  viewed  as  dysfunctional  with  regard  to  team 
performance"  (Klimoski  and  Mohammed,  1994). 

This  study  instead  proposes  that,  to  possess  appropriate  team  knowledge,  a  team  needs  to 
develop  “complementary  mental  models”  of  the  information  available  for  their  individual  and 
team  tasks.  Figure  3  represents  the  relationship  between  a  team’s  information  base,  task  relevant 
information,  shared  information  and  complementary  information.  This  is  similar  to  Cooke’s 
description  of  types  of  knowledge  (Cooke  et  al.,  2000)  but  distinctively  different  in  that  this 
representation  refers  to  the  information  available  within  a  team’s  domain  and  supports  the 
development  of  complementary  team  mental  models. 
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Figure  3.  Information  distribution  for  a  complementary  team  mental  model. 


An  aircraft  crew  is  an  example  of  a  domain  in  which  a  complementary  mental  model  is 
necessary.  Assuming  that  pilots  and  copilot/navigators  possess  a  similar  knowledge  base  (i.e., 
training),  in  some  instances,  the  pilot  and  the  copilot/navigator  might  need  to  have  some 
information  that  is  held  in  common.  However,  the  copilot/navigator  probably  won't  need  access 
to  all  of  the  information  the  pilot  needs  and  vice-versa.  Therefore  the  copilot/navigator  must  have 
some  information  that  is  complementary  to,  but  not  necessarily  identical  to,  the  pilot’s 
information.  Complementary  team  mental  models  are  extremely  beneficial  in  this  type  of  domain 
where,  although  each  team  member  fulfills  an  independent  role  on  the  team  as  well  as  an 
interdependent  role  as  team  member,  these  roles  may  switch  during  operations. 

This  study  hypothesizes  that  providing  each  team  member  with  the  information  relevant 
only  to  his/her  tasks  will  assist  in  the  establishment  of  complementary  team  mental  models.  A 
complementary  team  mental  model  is  defined  here  as  the  condition  in  which: 

•  Each  team  member  has  the  knowledge  necessary  to  conduct  his/her  tasks. 

•  Each  team  member  knows  which  information  is  available  to  the  other  team 
member  should  he/she  need  to  seek  it. 

•  Each  team  member  knows  which  information  is  needed  from  them  to  other  team 
members  and  when. 


This  study  supports  the  concept  of  a  “team  centered”  system  design  approach,  focused  on  a 


complementary  distribution  of  information  among  team  members  based  on  their  tasks.  This 
approach  will  naturally  promote  improved  team  coordination  by  aiding  team  members  in 
developing  a  complementary  team  mental  model.  Complementary  team  mental  models  can  be 
influenced  by  the  distribution  of  knowledge  between  team  members,  the  distribution  of 
information  in  the  environment,  or  a  combination  of  the  two  (see  Figure  4).  The  quadrants  in 
Figure  4  represent  the  levels  of  knowledge  and  information  distribution  as  described  below: 

•  Quadrant  I:  Team  member  knowledge  is  distributed  among  team  members.  The 
information  in  the  environment  is  common  to  all  team  members.  Based  on  the 
division  of  knowledge,  team  members  are  more  aware  of  their  specific 
information  requirements  and  can  identify  their  required  information  sources  in 
the  environment  more  readily. 

•  Quadrant  II:  Team  member  knowledge  is  distributed  among  team  members  and 
the  information  in  the  environment  is  complementarily  distributed.  The 
distribution  of  information  sources  in  the  environment  support  the  division  of 
knowledge  between  team  members,  and  performance  is  further  enhanced. 

•  Quadrant  III:  Team  knowledge  and  information  are  common  to  all  team  members. 
Team  members  must  rely  heavily  on  training,  operating  procedures,  checklists 
and  regulations  to  align  information  sources  of  the  environment  with  individual 
tasks  and  team  responsibilities. 

•  Quadrant  IV:  Team  knowledge  is  common  among  members,  yet  the  information 
in  the  environment  is  complementarily  distributed  between  team  members 
according  to  tasks  and  responsibilities. 


Information  Distribution 
Figure  4:  Knowledge  distribution  vs.  information 
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This  study  proposes  that  complementary  team  mental  models  are  enhanced  in  quadrants  I,  II, 
and  IV,  while  quadrant  ID  is  not  conducive  to  team  operations.  Teams  discussed  in  this  study 
rarely  possess  fully  distributed  knowledge;  generally  there  is  a  great  amount  of  knowledge 
overlap  between  members. 

Furthermore,  the  proposed  method  of  distributing  information  among  team  members  will 
provide  individual  crewmembers  with  a  more  accurate  task  relevant  mental  model  of  their 
environment.  The  formulation  of  complementary  team  mental  models  can  improve  team 
knowledge  and  team  performance  by  streamlining  team  processes,  helping  to  clarify  roles  and 
responsibilities,  individual  and  team  member  information  requirements,  and  improving  the 
efficiency  of  explicit  communications.  All  of  the  team  models  presented  in  this  section  result  in 
the  formulation  of  team  knowledge;  the  level  of  this  knowledge  is  manifested  through  team 
performance  as  shown  in  Figure  5.  By  efficiently  designing  the  environment  in  which  the  team 
processes  occur,  communication,  adaptiveness,  and  decision-making  also  should  become  more 
efficient. 


Figure  5.  Team  cognition  framework  (Cooke,  2002). 


Summary 

This  review  of  literature  makes  clear  the  following: 

•  Shared  mental  models  within  a  team  improve  performance. 

•  Traditionally,  the  premise  has  been  that  increasing  the  amount  of  information  that  is  shared 
between  team  members  will  naturally  improve  the  team’s  shared  mental  model. 

•  Previously,  no  empirical  evidence  demonstrated  that  providing  team  members  compatible  or 
complementary  information,  in  addition  to  common  knowledge,  improves  team  performance. 

•  Explicit  communication  is  related  to  the  presence  of  shared  mental  models  and  helps  to 
develop  accurate  shared  mental  models. 

•  Measurement  of  mental  models  is  difficult,  particularly  in  a  complex  environment;  therefore 
multiple  techniques  should  be  used  to  arrive  at  the  best  possible  assessment. 
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Some  recent  research  has  implied  that,  during  certain  situations  when  team  members  are 
presented  with  the  exact  same  information  within  a  problem  space,  other  factors  influence  their 
decision-making,  which  may  lead  team  members  to  different  conclusions.  This  anomaly  is  even 
evident  when  the  team  members  have  similar  training  and  backgrounds  (e.g.,  pilots).  Team 
members  may  perceive  the  same  cues  differently,  miss  information,  or  place  greater  importance 
on  different  information  (Fischer,  Orasanu,  and  Wich,  1995).  Redundancy  of  information 
between  team  members,  i.e.,  providing  everyone  with  all  information  available  in  a  problem 
space,  may  be  beneficial  in  some  situations  or  domains,  but  the  literature  suggests  that  there  may 
be  situations  when  there  is  greater  benefit  in  providing  complementary  task  relevant  information 
(i.e.,  providing  task  specific  information  to  individual  team  members  while  limiting  their  access 
to  non-task  relevant  information)  within  the  team.  That  is,  there  might  be  some  information 
overlap  required  among  team  members  but,  in  addition,  role-specific  complementary  information 
is  required  and  can  establish  a  complementary  team  mental  model  of  information  requirements 
within  the  operational  domain.  This  study  explores  the  development  of  complementary  team 
mental  models  and  the  distribution  of  information  within  a  teams  working  environment  in  two 
separate  domains  while  keeping  the  cognitive  tasks  relatively  consistent. 

Objective 

The  goal  of  this  research  was  to  further  the  understanding  of  crew  coordination  and  team 
performance  in  small  teams  under  different  information-sharing  conditions.  In  other  words,  the 
overarching  objective  of  this  study  was  to  examine  whether  providing  task  relevant  information 
to  individual  team  members  in  a  time  critical  environment,  while  limiting  their  access  to  non- 
relevant  information,  would  foster  complementary  team  mental  models  with  corresponding 
changes  to  team  process  and  performance. 

Determination  of  task  information  requirements 

Imagine  the  benefits  of  designing  the  information  distribution  within  a  team  to  naturally 
support  individual  and  team  tasks  and  encourage  efficient  communication  between  team 
members.  The  pursuit  of  such  a  design,  regardless  of  domain,  is  underpinned  by  the  accurate 
determination  of  information  sources  in  the  environment  and  by  team  information  requirements. 
This  chapter  proposes  that  information  sources  and  requirements  can  be  systematically  identified 
through  a  three  phase  analysis:  (1)  work  domain  analysis  of  the  information  available  in  the  work 
environment,  (2)  analysis  of  the  information  requirements  of  specific  tasks,  and  (3)  mapping 
information  sources  to  their  corresponding  tasks.  This  section  describes  a  unique  application  of 
two  separate  analytical  processes,  work  domain  analysis  and  hierarchical  task  analysis.  These 
processes  are  combined  to  link  information  sources  available  to  the  task  information 
requirements  of  a  team  operating  in  the  domain.  The  method  is  applicable  to  various  situations 
and  is  recommended  when  the  analyst  requires  a  descriptive  analysis  of  a  specified  task 
conducted  by  a  small  team  in  a  complex  domain.  This  method  is  discussed  in  general  and  then 
specifically  described  through  the  analysis  of  two  domains.  In  addition,  the  results  of  these 
examples  are  used  as  a  basis  for  two  experiments  discussed  in  subsequent  chapters. 
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Method 


Requirements  for  a  method 

Various  methods  exist  for  determining  information  requirements.  Aside  from  cognitive 
work  analysis,  human-centered  systems  design  (Rouse,  1991)  and  contextual  design  (Beyer  and 
Holtzblatt,  1998)  have  both  been  adopted  in  numerous  domains  and  have  appeared  to  be 
successful  relative  to  their  purposes  (Vicente,  1999).  These  methods  were  not  designed  to  be 
exclusive  analysis  tools  or  meant  to  replace  other  analysis  methods;  each  of  them  has  both 
positive  and  negative  aspects  when  applied  to  different  domains.  In  system  design,  particularly 
the  design  of  a  team  environment,  there  is  no  solution  that  applies  to  every  problem. 

The  basic  premise  of  this  team  centered  design  approach  is  that  the  system  should  support 
the  team  rather  than  vice  versa.  Too  often,  system  design  focuses  on  incorporating  new 
technology  or  displays  with  limited  consideration  of  how  a  team  will  actually  use  the  system. 

This  often  results  in  system  modifications  at  the  end  of  development,  or  even  after  completion, 
once  it  becomes  obvious  that  the  team  cannot  function  effectively.  If  retrofit  is  not  possible,  a 
team  may  resort  to  checklists,  procedures,  ad  hoc  practices,  or  additional  training  to  adapt  to  a 
system  that  does  not  support  their  tasks;  this  adds  complexity  and  the  team  becomes  less 
effective.  Emphasizing  the  team’s  needs  based  on  the  specified  tasks  that  will  be  conducted  in 
the  environment  can  lead  to  a  system  that  supports  team  operations. 

Norman  and  Draper  (1986)  and  Davis  (1989)  suggested  that  the  structure  of  the  presentation 
of  information  greatly  influences  a  user's  ideas  about  the  functions  of  the  system  or  an 
individual’s  mental  model.  Individual  mental  models  that  conform  to  the  actual  workings  of  the 
system  are  said  to  possess  the  quality  of  cognitive  compatibility  (Norman,  1983),  a  desired  state 
for  human-system  interaction.  Generally,  the  user's  understanding  of  the  system  is  diminished 
when  the  presentation  is  cluttered  and  disorganized,  and  when  it  does  not  directly  match  the 
functions  it  is  trying  to  support.  Logically,  team  interfaces  that  promote  high  cognitive 
compatibility  should  be  more  usable.  Two  similar  principles  that  address  the  design  of  displays 
for  human  use  are  the  proximity  compatibility  principle  (PCP)  and  the  ecological  compatibility 
principle  (ECP).  Fundamentally,  PCP  states  that  the  perceptual  characteristics  of  displays  should 
be  designed  to  be  compatible  with  the  cognitive  processes  used  by  operators  to  perform  a 
particular  task  (Wickens  and  Carswell,  1995).  ECP  suggests  that  the  design  process  should  begin 
by  ensuring  that  the  content  and  structure  of  the  interface  are  compatible  with  the  constraints  that 
actually  govern  the  process  to  be  controlled  (Vicente,  1997).  Both  of  these  principles  can  be,  and 
should  be,  extended  from  the  design  of  an  individual’s  interface  to  the  design  (i.e.,  organization) 
of  multiple  interfaces  for  teams.  PCP  is  based  on  an  information-processing  approach  to  human 
factors  focusing  on  how  the  human  will  use  this  information  in  the  display.  In  contrast,  ECP 
addresses  the  constraints  that  the  environment  places  on  the  processes  requiring  information. 

Extending  this  to  the  team  level  supports  the  use  of  a  task  analysis  (information-processing) 
method  combined  with  a  work  domain  analysis  (WDA)  (ecological  approach).  A  hierarchical 
task  analysis  focuses  on  the  observable  aspects  of  operator  behavior;  it  can  describe  the 
interactions  between  people  and  control  systems,  and  communication  requirements  between  team 
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members.  In  comparison,  analysis  of  the  work  domain  identifies  the  affordances  and  constraints 
of  the  environment  where  the  tasks  are  conducted.  The  combination  of  these  analyses  can 
identify  team-machine  interface  issues  that  arise  when  a  team  task  is  conducted  within  the 
environment,  and  insights  to  efficient  environmental  design  for  team  operations  can  be  gained. 

The  first  phase  of  this  method  is  a  work  domain  analysis  to  identify  the  structures  in  the 
environment  where  work  takes  place.  The  second  phase  conducts  a  separate  Hierarchical  Task 
Analyses  (HTA).  The  last  phase  maps  the  information  sources  available  in  each  domain  to  the 
individual  and  team  tasks  that  they  support. 

The  strengths  and  weaknesses  of  WDA  and  HTA,  discussed  below,  make  this  type  of 
analysis  most  appropriate  for  systems  where: 

•  The  domain  is: 

o  Supportive  of  team  operations;  i.e.,  generally,  tasks  require  more  than  one 
information  source  and  coordination  among  members  (Orasanu  and  Salas, 
1993;  Paris,  Salas,  and  Cannon-Bowers,  2000). 

o  Complex;  i.e.,  ill-structured  problems,  shifting,  or  competing  goals,  and  time 
stresses  with  severe  perceived  consequences  for  poor  performance  (Orasanu 
and  Connolly,  1993;  Rouse,  Cannon-Bowers,  and  Salas,  1992) 

•  The  tasks  are: 

o  Clearly  defined; 

o  Able  to  be  broken  down  into  goals,  tasks,  and  operations. 

Work  domain  analysis 


An  understanding  of  how  the  environment  affects  team  members  can  be  used  deliberately  to 
create  environments  to  suit  team  adaptation  (Carroll,  1991,  as  cited  in  Vicente,  1999). 
Accordingly,  a  thorough  description  of  the  work  domain  is  vital  to  system  design.  Work  domain 
analysis  is  a  widely  accepted  analysis  technique  that  has  been  successfully  applied  to  various 
domains,  including:  aviation,  command  and  control,  computer  programming,  engineering  design, 
information  retrieval,  medicine,  process  control,  and  workplace  design  (Vicente,  1999).  These 
work  domains  represent  the  system  being  controlled,  independent  of  workers,  automation,  event, 
task,  goal,  or  interface.  There  are  various  aspects  of  the  domain  that  can  be  the  focus  of  a  WDA; 
therefore,  the  purpose  of  the  analysis  must  be  defined.  The  purpose  of  this  general  analytical 
method  is  to  identify  sources  of  information  within  a  work  domain;  the  end  result  is  a  broad 
scope  of  the  information  sources  available  within  the  environment. 

The  representation  most  appropriate  for  this  analysis  is  the  ‘Abstraction-Decomposition 
Space  (ADS)’  (Vicente,  1999)  also  known  as  the  ‘Abstraction  Hierarchy  (AH)’  (Rasmussen, 
1985).  To  understand  the  work  environment,  and  later  its  relationship  to  the  task,  the  analyst 
must  look  at  all  levels  of  abstraction.  Each  level  represents  a  different  level  of  granularity  with 
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Figure  6.  Abstraction  hierarchy. 


which  the  analyst  can  represent  the  work  domain.  By  using  Rasmussen’s  abstraction  hierarchy 
(Rasmussen,  1985),  sources  of  information  can  be  organized.  The  analysts  can  identify 
environmental  constraints  and  affordances  and,  in  turn,  information  required  for  routine 
operations  and  unanticipated  situations. 

Work  domain  analysis  commonly  decomposes  a  system  along  two  dimensions,  as  shown  in 
Figure  6:  parts-whole  decomposition  and  means-ends  decomposition.  The  parts-whole 
decomposition  (from  left  to  right)  divides  the  system  into  a  hierarchy  of  progressively  smaller 
subsystems.  The  means-end  decomposition  (from  top  to  bottom)  divides  the  system  into 
hierarchical  levels  of  abstraction,  making  a  complete  representation  of  the  system  at  each  level. 
The  abstraction  hierarchy  consists  of  five  levels:  functional  purpose,  abstract  function,  general 
function,  physical  function,  and  physical  form  (Vicente,  1999).  The  organization  of  this  type  of 
hierarchy  answers  three  basic  questions  at  different  levels  of  the  hierarchy:  Why?  What?  and 
How?  Any  level  answers  the  question  “what”  the  element  is,  the  level  above  describes  “why” 
the  element  exists  and  the  level  below  describes  “how”  that  element  is  realized  (Vicente,  1999). 

WDA  is  designed  to  identify  environmental  constraints  and  affordances  within  the  system 
instead  of  identifying  the  cognitive  constraints  internal  to  the  system.  Likewise,  WDA  can 
identify  roles  of  agents  and  with  what  other  parts  of  the  system  those  agents  should  interact,  but  it 
will  not  specify  their  interactions.  This  type  of  analysis  assumes  that  workers  within  the 
environment  are  reasonably  familiar  with  the  domain;  and  outside  influences,  such  as  training, 
regulations,  or  procedures,  are  not  usually  included.  The  primary  weakness  of  WDA  is  that  it 
does  not  tell  the  workers  “what  to  do;”  it  only  describes  the  structure  of  the  system  they  will  be 
working  in.  This  is  why  a  work  domain  analysis  in  itself  is  not  sufficient  and  must  be  followed 
with  a  task  analysis. 
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Hierarchical  task  analysis 


The  term  “Task  Analysis”  can  be  applied  very  broadly  to  encompass  a  wide  variety  of 
human  factors  techniques.  Nearly  all  task  analysis  techniques  provide,  as  a  minimum,  a 
description  of  the  observable  aspects  of  operator  behavior  at  various  levels  of  detail,  together 
with  some  indications  of  the  structure  of  the  task.  These  have  been  referred  to  as  action  oriented 
approaches,  in  contrast  to  cognitive  approaches  that  focus  on  the  mental  processes  that  underlie 
observable  behavior  (Embrey,  2000).  This  research  focuses  on  the  action  oriented  approaches. 
Representations  developed  through  task  analysis  serve  as  reference  to  enhance  the  understanding 
of  the  human-system  involvement,  or  to  identify  particular  requirements  of  the  system.  There  are 
a  variety  of  task  description  techniques  available  to  the  analyst,  including:  charting  and  network 
techniques,  decomposition  methods,  HTA,  link  analysis,  operational  sequence  diagrams,  and 
timeline  analyses  (Kirwan  and  Ainsworth,  1992). 

For  the  subset  of  tasks  addressed  in  this  study,  the  use  of  HTA  is  advocated.  HTA  focuses 
on  the  observable  aspects  of  operator  behavior;  it  can  describe  the  interactions  between  team 
members,  and  communication  requirements  between  team  members.  HTA  was  introduced  by 
Annett  and  Duncan  (1967)  to  evaluate  an  organization’s  training.  It  is  an  approach  to  task 
analysis  that  prompts  the  analyst  to  establish  the  conditions  when  various  subtasks  should  be 
carried  out  in  order  to  achieve  the  system’s  goals;  these  task  components  are  then  graphically 
represented  using  a  structure  chart.  HTA  provides  analysts  with  great  flexibility  by  allowing 
them  to  determine  the  level  of  detail  to  which  the  hierarchy  is  developed. 

HTA  is  commonly  used  for  descriptive  analysis  of  a  task;  it  is  comprehensive  and  will  most 
accurately  identify  information  required  at  each  level  of  the  hierarchy  (see  Figure  7).  The  first 
level  relates  to  goals;  i.e.,  desired  states  of  systems  under  control  or  supervision  (make  a  cup  of 
tea);  the  second  to  tasks,  i.e.,  the  method  that  is  adopted  to  attain  the  goal;  and  the  third  to 
operations,  i.e.,  “Any  unit  of  behavior,  no  matter  how  long  or  short  its  duration,  or  how  simple  or 
complex  its  structure,  which  can  be  defined  in  terms  of  its  objective”  (Kirwan  and  Ainsworth, 
1992).  HTA  recognizes  the  responsibility  of  the  operator  to  plan  the  use  of  available  resources  to 
attain  a  given  goal  (Shepherd,  2001).  The  final  product  is  a  description  of  tasks  and  subtasks, 
identification  of  the  team  member  responsible  for  the  task,  and  corresponding  information 
requirements. 
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Figure  7.  Example  hierarchical  task  analysis. 


The  information  resources  required  must  be  identified  within  this  hierarchy  as  each  subtask 
is  developed.  For  instance,  step  5  in  Figure  7  states  that  the  subtask  is  to  “wait  4  or  5  minutes”; 
this  requires  a  chronometric  information  source  be  available  to  the  person  responsible  for  task  5. 
This  method  of  analysis  is  efficient  and  can  completely  describe  a  specified  task  along  with  the 
resources  required,  but  is  usually  very  narrow  in  scope  and  does  not  provide  the  user  the  ability 
to  adapt  to  unforeseen  contingencies  or  the  ability  to  recover  from  error. 

HTA  is  commonly  used  for  analysis  of  tasks  conducted  by  an  individual.  This  method 
augments  the  HTA  method  by  considering  a  team  component.  For  instance,  who  is  responsible 
for  the  task  and  their  information  requirements  are  included  in  the  hierarchy.  Taken  alone,  HTA 
would  not  be  a  sufficient  analysis  method.  This  method  becomes  flexible  to  contingencies  only 
when  combined  with  the  WDA  discussed  in  the  previous  section. 

Mapping  of  sources  to  requirements 


Once  the  sources  of  information  have  been  identified  through  the  work  domain  analysis  and 
information  requirements  established  based  on  the  hierarchical  task  analysis,  the  sources  must  be 
mapped  to  their  corresponding  task  requirements.  This  is  accomplished  by  comparing  the 
required  information  to  the  available  sources.  The  end  result  is  a  tabulation  of  which  information 
sources  each  team  member  requires  access  to  for  the  specified  task. 

Example  applications 

The  generalized  method  presented  above  was  applied  to  two  domains,  an  automobile  and  a 
military  helicopter,  and  a  specified  task,  team  navigation.  These  applications  both  demonstrate 
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the  method  and  identify  sources  of  information  to  distribute  in  a  complementarily  manner  during 
two  separate  experiments. 

Several  subject  matter  experts  were  consulted  during  the  work  domain  analysis  and  task 
analysis  for  both  domains,  to,  include:  active  and  retired  law  enforcement  experts  currently 
developing  training  programs  for  student  drivers  for  Road  Ready  Inc.  in  Duluth,  Georgia;  a 
military  helicopter  instructor  pilot  with  over  30  years  experience;  and  aero-medical  research 
psychologists.  Five  automobile  teams  were  observed  while  they  conducted  a  navigation  task  in  a 
fixed  base  automobile  simulator,  completing  over  20  scenarios  in  various  conditions.  For  the 
helicopter  domain,  ten  helicopter  teams  were  observed  during  40  flights.  Military  aircrew 
training  manuals,  helicopter  operator’s  manuals,  and  vehicle  GPS  navigation  training  websites 
provided  further  insights  into  both  analyses. 

Work  domain  analysis  examples 

Automobile  application  of  work  domain  analysis 

This  analysis  was  completed  for  the  “total  system,”  rather  than  one  taken  from  the 
perspective  of  a  driver,  navigator,  or  any  other  component  of  the  system,  and  the  purpose  of  the 
analysis  was  to  identify  information  sources  for  team  operations.  An  automobile  was  defined  as 
a  generic  midsize  vehicle  equipped  with  an  electronic  navigation  system.  The  system  boundaries 
were  defined  as  the  vehicle  and  the  environment  immediately  surrounding  the  vehicle.  A 
discussion  of  the  WDA  and  resulting  abstraction  hierarchy  (Figure  8)  follows. 


MEANS-ENDS 

RELATION 

AUTOMOBILE:  PROPERTIES  REPRESENTED 

Functional  Purpose 

Move  from  origin  to  destination.  Crew  survivability.  Environmental 
impact.  Provide  pleasure. 

Abstract  Function 

Conservation  of  energy.  Conservation  of  resources.  Flow  of 
information.  Balance  of  forces. 

;}\V  /.  *' ^ 

General  Function 

Process  of  distributing  information.  Process  of  moving  automobile. 
Process  of  launching  resources  (fuel,  water  air,  etc.).  Process  of 
coordinating  forces  (thrust,  weight,  and  drag).  Process  of 
environmental  control. 

Physical  Function 

Generating  signals/alerts.  Navigation.  Communication.  Driving 
input.  Crew  protection.  Speed  control.  Environmental  protection. 

i 

Physical  Form 

Driving  instrumentation,  system  health/status  indicators,  clock, 
composite  body,  air  bags,  foot  pedals,  steering  wheel,  engine, 
transmission,  seats,  side  panels,  windows,  window  washers, 
window  wipers,  tires,  headlamps,  horn,  radio,  heater,  air  conditioner, 
defroster,  internal  lights,  global  positioning  system,  cellular  phone, 
paint,  entertainment  system,  muffler,  catalytic  converter,  maps,  and 
procedures  guide. 

Figure  8.  Automobile  abstraction  hierarchy. 
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The  functional  purpose  level  corresponds  to  the  work  domain’s  purpose,  or  why 
the  system  exists.  There  were  four  functional  purposes  modeled  in  the  automobile  domain: 

Move  from  origin  to  destination,  crew  survivability,  environmental  safety,  and  pleasure.  This 
system  is  primarily  designed  to  safely  navigate  between  two  designated  points.  Crew 
survivability  describes  the  ability  of  the  system  to  protect  its  occupants  during  normal  and 
emergency  conditions.  Environmental  impact  involves  those  constraints  that  influence  the 
vehicles  impact  on  the  environment  due  to  various  types  of  pollution  produced  (i.e.,  air,  water, 
and  noise  pollution).  Driving  pleasure  involves  those  constraints  that  deal  with  comfort  of  the 
occupants  and  their  level  of  enjoyment  while  operating  in  the  domain.  The  level  to  which  a 
vehicle  satisfies  these  constraints  varies  between  specific  makes  and  models,  and  operators;  yet, 
the  functional  purposes  of  the  system  are  constant. 

The  abstract  level  defines  the  causal  laws  and  principles  underlying  the  function  of  the 
automobile;  they  may  need  to  be  minimized,  maximized  or  conserved  to  achieve  the  systems 
purpose  (Nadimian,  Griffiths,  and  Bums,  2002).  For  the  automobile’s  purpose  to  be  achieved, 
the  following  underlying  laws  and  principles  were  identified  at  the  abstract  level:  Conservation 
of  energy,  conservation  of  resources,  balance  of  forces,  and  flow  of  information.  Conservation  of 
energy  is  a  fundamental  concept  that  states  that  within  this  domain,  the  amount  of  energy  remains 
constant  and  energy  is  neither  created  nor  destroyed.  Energy  can  be  converted  from  one  form  to 
another  (potential  energy  can  be  converted  to  kinetic  energy),  but  the  total  energy  within  the 
domain  remains  fixed.  Conservation  of  resources  within  this  domain  seems  quite  obvious:  the 
amount  of  resources  remains  constant  as  resources  are  neither  brought  into  the  domain  nor  leave 
the  domain.  Most  moving  things  require  a  balance  of  forces,  since  most  moving  things  are 
affected  by  friction.  In  this  domain,  for  a  car  to  travel  at  a  steady  speed,  the  engine  and  drive  train 
must  supply  a  forward  force.  If  this  forward  force  is  stronger  than  the  combined  effects  of  wind 
resistance  and  friction  with  the  road,  the  car  speeds  up;  if  this  forward  force  is  weaker,  the  car 
slows  down.  The  flow  of  information  concerns  the  delivery  of  information  in  various  forms 
(analog,  digital,  audible,  etc.)  for  utilization  by  cognitive  mechanisms  within  the  domain. 

At  the  generalized  function  level,  the  processes  involved  in  achieving  the  functional 
purposes  of  the  automobile  by  influencing  the  constraints  at  the  abstract  level  were  modeled. 

Five  processes  were  identified  at  this  level:  Process  of  distributing  information,  physical  process 
of  moving  automobile,  process  of  launching  resources  (fuel,  water,  air,  etc.),  process  of 
coordinating  forces  (i.e.,  thrust,  weight,  and  drag),  and  process  of  environmental  control.  These 
processes  are  supported  by  and  achieved  through  the  physical  function  level  of  the  work  domain. 
This  level  of  the  analysis  describes  the  capabilities  of  the  physical  elements  of  the  system.  There 
were  eight  physical  functions  identified  for  this  system:  Generating  signals/alerts,  navigation, 
communication,  driving  input,  crew  protection,  speed  control,  environmental  control, 
environment  protection.  Generation  of  signals  and  alerts  describes  the  capabilities  of  a  number 
of  elements  at  the  physical  form  level  focused  on  the  presentation  of  information  to  operators. 
Navigation  addresses  functions  used  to  assist  the  operator  in  navigation.  Communication 
addresses  the  functions  required  to  talk  to  other  members  within  the  domain  and  receive  and  send 
information  outside  the  domain.  Driving  input  and  speed  control  address  those  functions  that 
accelerate,  decelerate  and  steer  the  vehicle,  along  with  any  constraints  or  controls  on  velocity 
limits.  Environmental  control  deals  with  the  comfort  and  conduciveness  of  the  environment  to 
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operations  being  conducted,  while  environmental  protection  deals  with  the  environment  outside 
of  the  vehicle. 

Lastly,  the  physical  form  level  was  modeled.  At  this  level,  the  most  concrete  level,  the 
configuration  of  the  elements  in  the  domain  that  supports  the  functions  described  above  were 
identified.  The  elements  in  this  level  are  self-explanatory  and  are  displayed  in  Figure  8. 

Military  helicopter  application  of  work  domain  analysis 

The  same  method  used  for  the  automobile  domain  was  applied  to  a  military  helicopter 
domain.  This  analysis  also  was  completed  for  the  “total  system,”  rather  than  one  taken  from  the 
perspective  of  a  pilot,  navigator,  or  any  other  component  of  the  system.  The  military  helicopter 
considered  here  is  a  modernized  helicopter  with  an  integrated  electronic  navigation  system.  The 
system  boundaries  were  defined  as  the  aircraft  cockpit  and  the  environment  immediately 
surrounding  the  aircraft.  A  discussion  of  the  abstraction  hierarchy  analysis  follows;  a  summary 
of  the  findings  is  shown  in  Figure  9. 


MEANS-ENDS 

RELATION 

HELICOPTER:  PROPERTIES  REPRESENTED 

?  Functional  Purpose 

Move  from  origin  to  destination.  Crew  survivability.  Environmental 
impact.  Provide  ordnance  delivery  platform.  Provide  observation 
platform. 

Abstract  Function 

Conservation  of  energy.  Conservation  of  resources.  Flow  of 
information.  Balance  of  forces. 

Process  of  distributing  information.  Process  of  moving  aircraft. 
Process  of  launching  resources  (fuel,  water  air,  etc.).  Process  of 
coordinating  forces  (thrust,  drag,  lift,  and  weight).  Process  of 
targeting.  Process  of  environmental  control. 

Physical  Function 

Generating  signals/alerts.  Navigation.  Communication.  Flying 
input.  Crew  protection.  Speed  control.  Environmental  control. 
Environmental  protection.  Weapons  control. 

B 

Flight  instrumentation,  system  health/status  indicators,  clock, 
composite  body,  flight  controls,  engine,  transmission,  main  rotor 
blades,  tail  rotor,  seats,  armored  side  panels,  wind  screens,  wind 
screen  wipers,  wheels,  landing  lights,  radios,  heater,  air  conditioner, 
defroster,  internal  lights,  global  positioning  system,  fire  control 
computer,  fire  control  panel,  targeting  sight,  signature  reducing 
paint,  maps,  environmental  indicators,  voice  communication  radios, 
(FM,  VHF,  UHF),  navigation  radios  (ADF,  VOR),  aircraft  survivability 
equipment  (IR  suppressor,  AN-ALQ  136),  procedures  guide  and 
checklist,  and  vision  enhancement  devices. 

Figure  9.  Helicopter  abstraction  hierarchy. 
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There  were  Two,  functional  purposes  modeled  in  the  helicopter  domain:  Move  from  origin  to 
destination,  crew  survivability,  environmental  impact,  provide  ordnance  delivery  platform,  and 
provide  observation  platform.  Similar  to  the  automobile  domain,  this  system  is  primarily 
designed  to  safely  navigate  between  two  designated  points.  Crew  survivability  and 
environmental  were  conceptually  the  same  as  previously  discussed.  Two  additional  functional 
purposes  of  the  system  were  to  provide  both  an  ordnance  delivery  and  observation  platform. 
These  capabilities  vary  significantly  between  aircraft  types,  but  are  often  essential  functions  of 
the  total  system. 

For  the  helicopter’s  purpose  to  be  achieved,  the  following  underlying  laws  and  principles 
were  identified  at  the  abstract  level :  Conservation  of  energy,  conservation  of  resources,  balance 
of  forces,  and  flow  of  information.  These  are  conceptually  identical  to  the  automobile  domain. 
The  additional  force  of  lift  was  added  to  the  balance  of  forces  for  this  domain. 

At  the  generalized  function  level,  the  processes  involved  in  achieving  its  functional  purposes 
were  modeled.  Six  processes  were  identified  at  this  level:  Process  of  distributing  information, 
physical  process  of  moving  automobile,  process  of  launching  resources  (fuel,  water,  air,  etc.), 
process  of  coordinating  forces  (i.e.,  thrust,  weight,  and  drag),  process  of  environmental  control, 
and  process  of  targeting.  The  only  process  that  is  different  in  the  helicopter  domain  is  the 
process  of  targeting  which  refers  to  the  processes  used  by  the  aircraft  and  the  team  to  locate  and 
identify  targets  outside  of  their  environment. 

The  physical  function  level  of  the  work  domain  describes  the  capabilities  of  the  physical 
elements  of  the  system.  There  were  nine  physical  functions  identified  for  this  system:  Generating 
signals/alerts,  navigation,  communication,  driving  input,  crew  protection,  speed  control, 
environmental  control,  environment  protection,  and  weapons  control.  The  control  of  weapon 
systems  was  the  only  physical  function  particular  to  the  helicopter  domain;  this  concerns  the 
human  machine  interfaces  integral  to  weapon  systems  operations. 

Summary  of  WDA:  automobile  and  helicopter 

The  diversity  within  each  environment  at  the  functional  purpose  level  reaffirms  the 
complexity  of  the  domains  and  differentiates  them  from  other  domains.  Only  a  small  number  of 
domains  must  combine  efficient  and  timely  movement,  survivability,  and  safety  measures  in 
order  to  meet  their  functional  purpose.  As  well  as  providing  a  means  to  identify  information 
sources  to  manipulate  during  experimentation,  these  AH  analyses  highlighted  some  similar 
constraints  and  affordances  between  these  outwardly  different  complex  environments;  many  of 
the  levels  were,  in  essence,  uniform  across  the  two  domains.  It  was  evident  that  the  first  four 
levels  of  abstraction  were  very  similar  between  domains;  the  domains  began  to  diverge  at  the 
physical  form  level.  In  Figure  10  cells  highlighted  in  green  (medium  gray  in  black  and  white 
[B&W])  depict  elements  of  the  hierarchy  that  had  a  direct  mapping  from  one  domain  to  the 
other;  red  (dark  gray  in  B&W)  indicates  that  no  mapping  existed.  The  majority  of  the  functions 
or  purposes  that  did  not  map  primarily  dealt  with  the  two  functional  purposes  specific  to  the 
helicopter  (i.e.,  observation  platform  and  weapons  platform)  and  the  one  functional  purpose 
specific  to  the  automobile  (i.e.,  pleasure). 
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The  abstraction  provides  an  instrument  to  compare  functions  across  domains.  For  example, 
although  the  navigation  system  supports  weapons  control  for  a  helicopter,  it  also  supports  many 
of  the  same  functions  that  appear  in  an  automobile  abstraction  hierarchy:  Information  collection, 
process  of  moving  the  vehicle,  and  environmental  control.  As  the  abstraction  hierarchy  is 
descended,  the  divergence  of  the  environments  occurs  the  most  at  the  physical  form  level,  the 
most  concrete  level  of  the  hierarchy.  While  the  physical  forms  required  to  support  the  functions 
and  purposes  outlined  in  the  AH  are  clearly  different  between  these  domains,  their  functional 
structures  impose  similar  constraints  and  affordances  to  the  actors  operating  in  each  of  these 
environments,  which  may  lead  to  similar  behavior  patterns  of  teams  operating  in  these  domains. 

Ecological  Interface  Design  (EID)  techniques,  such  as  the  AH,  are  based  on  the  concept  that 
the  constraints  of  the  environment  must  be  explicitly  analyzed  to  enable  the  direct  perception  of 
goal  relevant  properties  of  the  environment  (Rasmussen  and  Vicente,  1989;  Vicente,  2002; 
Vicente  and  Rasmussen,  1992).  Using  this  method  helped  to  identify  sources  of  information 
available  to  agents  operating  in  these  environments.  These  sources  of  information  identified 
were  the  basis  for  determining  main  parameter  levels  in  the  experiments  conducted  in  this  effort. 

Hierarchical  task  analysis  example 

Automobile  and  helicopter  application  of  task  analysis 

In  many  domains,  the  timing  and  order  of  tasks  is  important,  especially  when  the  tasks  are 
serial  in  nature.  The  tasks  and  sub-tasks  particular  to  team  navigation  are  not  serial  tasks; 
actually,  they  must  be  conducted  in  parallel  to  each  other.  For  instance,  for  the  first  sub-goal 
level  in  Figure  12  the  team  must  maintain  vehicle  control  while  navigating;  this  implies  general 
team  duties  must  be  performed  while  the  team  obeys  traffic  laws.  All  of  these  sub-goals  are 
executed  in  parallel.  Therefore,  although  one  of  the  products  normally  developed  during  a 
hierarchical  task  analysis  is  a  series  of  plans  that  state  conditions  that  specify  when  each  of  a  set 
of  sub-goals  should  be  carried  out  (Kirwan  and  Ainsworth,  1992),  this  was  not  done  for  the 
navigation  task.  The  parallel  nature  of  navigation  adds  to  the  complexity  of  both  domains. 
Figures  12  and  13  represent  the  HTA  for  aircrew  and  automobile  team  navigation. 

Using  this  method  helped  to  identify  tasks  that  were  similar  to  both  domains.  Figure  13, 
using  the  helicopter  domain  as  the  reference  domain,  highlights  similarities  in  the  task  between 
the  two  environments  using  the  following  color  scheme: 

•  Green  (medium  gray  in  B&W):  Those  tasks/sub-tasks  of  the  hierarchy  that  had  a  direct 

mapping  from  one  domain  to  the  other. 

•  Yellow  (light  gray  in  B&W):  Although  there  was  no  direct  mapping  between  domains, 
there  was  a  strong  similarity.  For  example,  instead  of  conforming  to  Air  Traffic  Control 
(ATC)  rules  and  regulations  (helicopter  domain),  agents  in  the  automobile  domain  are 
required  to  obey  the  rules  of  the  road.  These  two  sub-goals  are  analogous. 

•  Red  (dark  gray  in  B&W):  No  mapping  existed. 

Identifying  similar  task  requirements  assisted  in  the  development  of  experimental  task 
requirements. 
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Mapping  of  sources  to  requirements:  example 


The  HTAs  were  developed  using  the  top-level  goal  “move  from  point  A  to  point  B.”  For 
this  example,  a  specific  sub-task  was  selected  that  could  be  performed  in  both  domains  during 
experimentation.  Task  selection  was  accomplished  utilizing  the  similarities  identified  in  Figure 
13.  Radio  aided  navigation  could  not  be  performed  in  the  automobile  domain  and,  due  to 
limitations  of  the  simulator  used  for  the  automobile  experiment,  electronically  aided  navigation 
could  not  be  performed  either.  Additionally,  due  to  the  goals  of  this  research,  mission  planning 
was  not  incorporated  in  either  domain.  All  other  sub-tasks  were  incorporated  into  experimental 
scenarios.  The  primary  tasks  identified  for  experimentation  were  (1)  navigation  by  pilotage  and 
dead  reckoning  (helicopter),  and  (2)  conventional  map  navigation  (automobile). 
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Automobile 


Figure  10.  Similarity  between  work  domains. 
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Crew  protection 


Figure  11.  Helicopter  team  navigation  HTA. 
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Figure  12.  Automobile  team  navigation  HTA. 
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speed  changes  necessary  to 
navigate  the  desired  location 
2.1.5  The  D  will  announce 
significant  checkpoints  to 
assist  m  navigation 


Figure  13.  Domain  similarities  in  team  navigation  HTA. 
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airspeed  changes  necessary  to 
navigate  the  desired  course 
2.1.5  The  P  will  announce  significant 
terrain  features  to  assist  in 
navigation 


Once  the  task  was  selected,  the  physical  forms  that  supported  this  task  were  identified. 
Figures  8  and  9  identify  sources  of  information  available  in  each  domain  that  support  these  tasks; 
these  are  combined  in  Figure  14.  Based  on  the  task  conditions  established  for  the  experimental 
scenarios,  the  requirements  were  reduced;  e.g.,  all  scenarios  were  conducted  in  the  daylight  and 
no  scenarios  incorporated  rain,  hail,  snow,  or  environmental  issues  that  could  be  diagnosed  by 
the  team  (excluding  Inadvertent  Instrument  Meteorological  Conditions  in  the  helicopter  domain). 
Additionally,  participants  of  the  experiments  would  not  switch  duty  positions;  therefore,  the 
driver/pilot  would  always  have  the  driving/flying  controls.  The  following  physical  forms  were 
either  not  available  for  experimental  purposes  or  not  considered  essential  for  operating  in  the 
automobile  domain  under  the  stated  conditions:  Window  washers,  window  wipers,  headlamps, 
radio,  internal  lights,  global  positioning  system,  and  cellular  phone.  Likewise,  the  following 
elements  were  not  considered  in  the  helicopter  domain:  Windscreen  wipers,  landing  light, 
targeting  sight,  environmental  indicators,  navigation  radios  (ADF,  VOR),  and  vision 
enhancement  devices. 

Finally,  using  the  task  descriptions  presented  in  Figures  1 1  and  12,  the  information  sources 
identified  in  Figure  14  were  allocated  to  the  corresponding  team  member  that  would  perform  the 
task.  Figure  15  shows  the  information  sources  for  each  team  member  for  each  domain. 


AUTOMOBILE 

HELICOPTER 

Physical 

Forms 

Supporting 

Navigation 

Tasks 

Driving  instrumentation,  system 
health/status  indicators,  driving 
controls,  (foot  pedals,  steering 
wheel),  windows,  window  washers, 
window  wipers,  headlamps,  radio, 
internal  lights,  global  positioning 
system,  cellular  phone,  and  maps. 

Flight  instrumentation,  system 
health/status  indicators,  clock,  flight 
controls,  wind  screens,  wind  screen 
wipers,  landing  lights,  radios,  internal 
lights,  global  positioning  system, 
targeting  sight,  maps,  environmental 
indicators,  voice  communication 
radios,  (FM,  VHF,  UHF),  navigation 
radios  (ADF,  VOR),  and  vision 
enhancement  devices. 

Reduced 

Physical 

Forms 

Supporting 

Navigation 

Tasks ■ 

Driving  instrumentation,  system 
health/status  indicators,  clock, 
driving  controls,  windows,  internal 
lights,  and  maps. 

Flight  instrumentation,  system 
health/status  indicators,  clock,  flight 
controls,  wind  screens,  radios,  internal 
lights,  global  positioning  system, 
maps,  and  voice  communication 
radios,  (FM,  VHF,  UHF). 

Figure  14.  Physical  forms  that  support  navigation. 
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Diver/Pilot 

Navigator 
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Ffysical  Farms  Rsquredfor  Navigation 

f  AJorrcbile 

Hsliocpter 

Diving  Instnmerts 

Flicft  Instnmerts 

WndcwViaA/ 

WncfccreenView 

S^teml-fealtfVStstus  Indcators 

S^emHealtlVStetus  Indcators 

dock 

dock 

IVfeps 

fvfeps 

Qcbal  Positioning  System 

Internal  Hefts 

Vcioe  ccrrmnication  Ftedos 

Internal  Hefts 

Figure  15.  Physical  forms  required  by  team  member. 


Summary 

This  section  has  (1)  established  that  a  requirement  exists  for  a  method  to  identify 
information  sources  and  align  them  with  task  requirements  within  a  complex  domain  supporting 
team  operations,  and  (2)  proposed  a  method  that  identifies  these  sources  and  requirements 
through  a  unique  three-phase  analysis: 

•  Analysis  of  the  work  domain, 

•  Analysis  of  a  specific  task  within  domain,  and 

•  A  mapping  of  the  information  sources  to  their  supported  tasks. 

The  combination  of  methods  described  in  this  chapter  offer  a  flexible  method  for  a  variety  of 
domains  and  task  combinations.  The  strengths  and  weaknesses  of  WDA  and  HTA  make  this 
type  of  analysis  most  appropriate  for  systems  where  the: 

•  The  domain  is: 

o  Supportive  of  team  operations;  i.e.,  generally,  tasks  require  more  than  one 
information  source  and  coordination  among  members  (Orasanu  and  Salas, 
1993;  Paris,  Salas,  and  Cannon-Bowers,  2000). 
o  Complex;  i.e.,  ill-structured  problems,  shifting,  or  competing  goals,  and  time 
stresses  with  severe  perceived  consequences  for  poor  performance  (Orasanu 
and  Connolly,  1993;  Rouse,  Cannon-Bowers,  and  Salas,  1992). 
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•  The  tasks  are: 

o  Clearly  defined; 

o  Able  to  be  broken  down  into  goals,  tasks,  and  operations. 

Experiment  I:  fixed  base  automobile  driving  simulator 

The  overarching  objective  of  this  study  was  to  examine  whether  providing  task  relevant 
information  to  individual  team  members  in  a  time  critical  environment,  while  limiting  their 
access  to  non-relevant  information,  would  foster  complementary  team  mental  models  with 
corresponding  changes  to  team  process  and  performance.  Furthermore,  this  method  of 
distributing  information  among  team  members  would  provide  individual  team  members  with  a 
more  accurate  “task  relevant”  mental  model  of  their  own  environment.  The  central  proposition 
of  this  study  is  that  providing  specific  task  relevant  information  to  individual  team  members  in  a 
time  critical  environment,  while  limiting  their  access  to  non-relevant  information,  will  change 
team  coordination  and  develop  complementary  team  mental  models.  Furthermore,  this  method 
of  distributing  information  among  team  members  will  provide  individual  team  members  with  a 
more  accurate  “task  relevant”  mental  model  of  their  environment. 

Note  that  this  study  will  employ  the  terms  nominal  and  off-nominal  to  describe  experimental 
conditions.  A  condition  characterized  as  nominal  implies  that  no  experimental  changes/inputs 
were  imposed  on  the  team  which  altered  their  planned  navigational  runs.  During  off-nominal 
conditions,  teams  were  forced  to  alter  their  planned  runs  due  to  unexpected  events  imposed  by 
the  researchers. 

Specific  hypotheses  that  investigate  this  central  proposition  are: 

•  Hypothesis  1 :  Team  performance,  during  nominal  conditions,  will,  generally,  remain 
constant  when  information  is  complementary  when  compared  to  performance  in  a 
normal  automobile  cockpit  configuration. 

•  Hypothesis  2:  Team  performance  during  off-nominal  conditions  will  improve  when 
information  is  complementary  when  compared  to  performance  in  a  normal 
automobile  cockpit  configuration. 

•  Hypothesis  3:  Explicit,  task  relevant,  verbal  communications  will  increase  when 
information  is  complementary  when  compared  to  communications  in  a  normal 
automobile  cockpit  configuration. 

•  Hypothesis  4\  The  team’s  task  relevant  anticipation  ratio  will  increase  when 
information  distribution  is  complementary  when  compared  to  operations  in  a  normal 
automobile  cockpit  configuration. 

•  Hypothesis  5:  Individual  workload  ratings  will  remain  constant  between  information 
levels  regardless  of  cockpit  configuration. 

Method 

This  experiment  was  conducted  at  Safe  Drive  Technologies,  Inc.,  located  in  Duluth, 

Georgia,  using  students  from  the  Georgia  Institute  of  Technology  as  participants.  The  main 


31 


parameter  was  the  complementariness  of  task  specific  information  available  to  team  members. 
During  the  experiment  data  were  collected  concerning  team  communications,  team  workload, 
decision-making,  and  performance  while  the  participants  conducted  a  navigation  task  in  a  time 
critical  situation.  Each  team  member  assumed  a  different  role  in  the  team,  either  the  driver  or  the 
navigator;  they  maintained  their  assigned  role  throughout  the  entire  experiment  (i.e.,  there  was  no 
role  switching). 


Participants 

Participants  were  24  students  from  the  Georgia  Institute  of  Technology  tested  in  pairs  with 
the  following  characteristics: 

•  All  possessed  a  current  U.S.  driver’s  license. 

•  Total  driving  experience  ranged  from  3.5  to  21  years;  average  experience  was  7.8  years. 

•  Participants’  ages  ranged  from  19-37;  average  age  was  24  years. 

Experiment  apparatus 

•  The  GE  Capital  I-Sim's  PatrolSim™  compact,  fixed  base  driving  simulator  was  used  for 
this  study  (Figures  16  and  17)  and  is  further  described  in  Appendix  A).  This  provided  an 
interactive  environment  in  which  team  performance  could  be  observed  while  certain 
parameters  were  controlled. 

•  Foggles  are  manufactured  glasses  that  are  used  as  a  tool  during  instrument  training  to 
limit  the  pilot's  field  of  vision.  This  assists  the  learning  process  by  encouraging  the  pilot 
to  fly  with  only  the  help  of  the  flight  instruments.  In  this  case,  the  open  area  in  the 
glasses  was  just  large  enough  to  allow  the  navigator  to  have  a  full  view  of  the  driving 
instructions  without  a  view  out  of  the  windows  of  the  automobile.  Foggles  are  light, 
comfortable,  and  easy  to  wear;  they  are  designed  like  regular  glasses. 

•  A  video  camera,  which  also  recorded  audio,  was  temporarily  mounted  to  record  team 
interaction  throughout  the  experiment.  All  data  collection  runs  had  video  and  sound  of 
the  participants  recorded. 

•  Maps  of  the  simulated  city  used  for  the  experiment  were  provided  to  the  team.  Driving 
directions  from  the  starting  point  to  the  team’s  destination  were  also  provided.  A  sample 
map  is  shown  in  Figure  18.  The  complete  set  of  maps  used  for  the  experiment  is  in 
Appendix  A. 
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Figure  16.  GE  Capital  I-Sim's  PatrolSim™,  View  I. 


Figure  17.  GE  Capital  I-Sim's  PatrolSim™,  View  II. 


33 


Experiment  procedure 
Introductory  briefing 

•  Sought  informed  consent  from  participants:  A  copy  of  the  consent  form  is  contained  in 
Appendix  A. 

•  Explained  the  experiment. 

•  Detailed  the  schedule  of  events. 

•  Oriented  participants  with  the  facilities. 

•  Collected  demographical  data  from  participants. 
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Training  run 


The  training  run  ensured  that  each  participant  was  familiar  with  the  feel  of  the  simulator,  the 
instrument  displays,  and  the  area  of  operation. 

Data  runs 

Participants  conducted  five  navigation  runs  during  which  data  were  collected: 

•  Four  nominal  data  runs  (no  diversions ).  During  these  runs,  the  driver  and  navigator  were 
placed  in  a  series  of  scenarios  that  were  generally  the  same.  They  started  at  a  specified 
location  and  were  asked  to  navigate  to  a  destination;  the  information  provided  to  team 
members  was  varied  during  these  runs.  The  effects  of  pre-planning  were  not  being  tested; 
therefore,  teams  were  not  allowed  time  to  plan  out  a  strategy  before  the  start  of  each  run. 
The  runs  each  took  approximately  10  minutes,  with  a  break  given  between  them. 
Immediately  following  each  of  the  navigation  runs  participants  completed  surveys 
regarding  their  workload,  performance,  etc. 

•  One  off-nominal  data  run  (with  diversions).  Participants  were  asked  to  partake  in  a  fifth 
data  run.  This  run  was  generally  the  same  as  the  previous  four  scenarios  except  a  planned 
detour  was  included.  This  detour  was  designed  to  divert  the  team  from  their  given  route 
and  directions.  The  team  was  expected  to  formulate  a  plan,  navigate  back  onto  the  route, 
and  complete  the  run. 

End  of  experiment  survey 

Following  the  complete  experiment,  individuals  completed  a  final  survey  concerning  overall 
themes  of  the  experiment.  The  entire  procedure  lasted  approximately  2.5  hours  for  each  team; 
this  included  introduction  briefing,  simulator  familiarization,  training  runs,  data  runs,  and  all 
surveys.  Participants  were  free  to  request  a  break  at  any  time. 

Task  description 

This  experiment  was  conducted  in  an  automobile  simulator  provided  by  ROAD  READY 
Inc.  and  was  designed  to  collect  data  concerning  team  coordination,  decision  making  and 
performance  while  conducting  a  navigation  task  under  a  time  critical  situation.  Each  team 
member  assumed  a  different  role  in  the  team,  either  the  vehicle  driver  or  navigator.  The  team 
was  given  a  map  and  textual  directions  to  various  locations  within  the  simulated  environment. 
They  were  asked  to  navigate  in  two  specific  conditions  (nominal  and  off-nominal). 

Under  nominal  conditions  participants  were  asked  to  conduct  four  navigation  runs.  During  these 
runs,  the  driver  and  navigator  were  placed  in  a  series  of  scenarios  that  were  functionally 
equivalent.  They  started  at  a  specified  location  and  were  asked  to  navigate  to  a  destination;  the 
information  provided  to  team  members  was  varied  during  these  runs.  Teams  were  asked  to 
maintain  control  of  the  vehicle,  obey  common  traffic  laws  (e.g.,  speed  limit,  stop  lights,  right  on 
red,  one  way  streets,  etc.),  identify  a  speed  zone,  and  navigate  to  the  location  given  on  their  map 
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using  the  route  provided.  Teams  were  expected  to  conduct  individual  tasks  (i.e.,  maintain 
vehicle  control),  team  tasks  (i.e.,  navigation),  coordinate  their  actions  as  a  team,  and 
communicate  their  actions,  intentions  and  requests  for  information. 

The  off-nominal  run  consisted  of  one  experimental  run  similar  to  the  previous  four, 
however,  unexpectedly  to  the  team,  a  roadblock  required  a  detour  from  the  designated  route. 

This  run  thus  assessed  team  responses  to  an  unexpected,  “off-nominal”  scenario;  the  detour  was 
designed  to  divert  the  team  from  their  given  route  and  directions.  The  roadblock  could  be 
overcome  in  a  variety  of  ways  and  the  optimal  solution  was  not  clear.  In  addition  to  the 
expectations  for  the  nominal  condition,  teams  were  expected  to  formulate  a  plan,  navigate  back 
onto  the  route,  and  complete  the  navigation  task. 

Each  run  commenced  when  the  driver  turned  the  ignition  key  and  started  the  vehicle.  To 
avoid  any  pre-planning,  the  team  members  had  one  minute  to  review  the  directions  and  then  were 
asked  to  start  the  simulation.  The  driver  and  navigator  were  only  allowed  to  use  verbal 
communication;  i.e.,  gestures  such  as  pointing  out  the  window  or  on  the  map  were  not  allowed. 

If  teams  made  a  mistake  and  deviated  off  the  route,  they  were  asked  to  get  back  on  the  route  as 
soon  as  possible.  The  experiment  ended  when  the  team  arrived  at  their  destination  and  turned  the 
vehicle  off. 

Experimental  design 

The  overall  study  consisted  of  two  experiments  run  sequentially;  participants  were  unaware 
that  there  were  two  experiments.  The  first  experiment  design  consisted  of  four  runs  that 
examined  performance  under  nominal  conditions,  in  which  the  four  levels  of  information 
distribution  were  varied.  For  these  first  four  runs,  a  Greco-Latin  Square  design  was  used  to 
minimize  any  effects  of  the  scenario,  order,  and  training,  and  to  isolate  the  effects  of  the 
independent  factors.  In  these  four  cases,  the  teams  were  presented  with  ‘nominal’  scenarios  in 
which  they  could  navigate  without  deviation  from  the  route  to  their  destination.  Specifically, 
there  were  no  inconsistencies  between  the  route  on  the  map  and  the  ability  to  follow  the  route  in 
the  simulator  (e.g.,  there  were  no  road  blocks,  detours,  etc).  The  second  experiment  consisted  of 
one  additional  experimental  run  similar  to  the  previous  four  with  the  addition  of  a  detour.  The 
information  distribution  conditions  were  balanced  and  treated  as  a  between-subjects  variable  for 
this  run.  The  same  scenario  was  used  for  all  teams. 

Independent  factors 

During  the  experiment,  variations  in  measures  were  observed  as  the  investigators  presented 
different  levels  of  information  distribution  to  the  participants.  Two  independent  factors 
determined  the  distribution  of  information  within  the  team.  These  independent  factors  were:  (1) 
whether  the  driver  has  access  to  the  map  used  for  navigation  and,  (2)  whether  the  navigator  has 
access  to  the  “out-of-window”  view. 
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Figure  19  shows  a  matrix  that  was  used  to  determine  levels  of  information.  The  two  by  two 
matrix  resulted  in  four  levels  of  information  distribution.  These  levels  were  designed  by  the 
investigator  to  change  from  non-complementary  (Level  I)  to  fully  complementary  (Level  IV)  and 
are  described  in  detail  below. 
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Figure  19.  Information  level  matrix. 


Level  I  (Non-Complementary) 

The  driver  and  navigator  were  both  provided  with  identical  information,  (i.e.,  textual 
directions  to  the  destination  and  a  map  of  the  area  with  the  route  highlighted,  and  both  had  an 
unrestricted  out-of-the-window  view). 

Level  II  (Partially  Complementary) 

Only  the  navigator  was  provided  with  textual  directions  to  the  destination  and  a  map  of  the 
area  with  the  route  highlighted.  The  navigator  was  not  allowed  to  show  the  map  or  the  textual 
directions  to  the  driver  and  instead  needed  to  provide  the  driver  with  steering  commands.  Both 
the  driver  and  the  navigator  had  an  unrestricted  out-of-the-window  view. 

Level  III  ( Partially  Complementary) 

The  driver  and  navigator  were  both  provided  with  the  textual  directions  to  the  destination 
and  a  map  of  the  area  with  the  route  highlighted.  The  navigator  wore  plastic  eyeglasses 
(Foggles)  that  were  clear  only  in  his  or  her  line  of  sight  to  a  map  held  on  the  lap,  but  were  opaque 
around  that  narrow  field  of  view,  thus  obscuring  his  or  her  sight  of  the  out-of-the-window  view. 

Level  IV  (Fully-Complementary) 

Only  the  navigator  was  provided  with  textual  directions  to  the  destination  and  a  map  of  the 
area  with  the  route  highlighted.  The  navigator  wore  Foggles. 

Dependent  Factors 

Dependent  Measures  were  categorized  into  three  groups:  Performance,  Workload,  and 
Communication. 
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Performance 

Performance  under  nominal  conditions  was  assessed  using  five  measures  and  two  collection 
sources.  Speed  violations,  lane  violations,  and  hard  decelerations  were  obtained  through  a 
driver’s  assessment  report  compiled  by  the  simulator.  These  were  recorded  as  the  sum  of 
individual  occurrences  of  each  violation.  The  experimenter  observed  navigation  errors,  and 
speed  zone  recognition.  Navigation  errors  included  both  committed  navigation  errors  and  near 
navigation  errors.  Committed  errors  were  instances  in  which  the  team  navigated  off  of  the  given 
route;  near  errors  were  instances  where  the  navigator  gave  the  wrong  instructions  or  the  driver 
attempted  to  navigate  off  the  given  route.  Speed  zone  recognition  was  recorded  as  ‘yes’  or  ‘no’ 
depending  on  whether  the  team  identified  a  posted  reduction  in  speed  (via  speed  limit  sign) 
within  the  scenario;  speed  zone  reductions  were  limited  to  one  per  experimental  run. 

Performance  under  “off-nominal”  conditions  was  measured  by  team  Decision  Time  (DT)  when 
faced  with  an  unexpected  detour  and  the  Total  Time  (TT)  it  took  the  team  to  get  back  on  the 
designated  route,  this  indicated  the  quality  of  their  decision. 

Workload 

Workload  was  measured  through  the  use  of  the  NASA  Task  Load  Index  (TLX)  (Hart  and 
Staveland,  1988).  Six  measures  of  workload  were  collected  via  the  NASA  TLX  subjective  rating 
sub-scales:  mental  demand,  physical  demand,  temporal  demand,  performance,  effort,  and 
frustration.  Team  members  scored  these  measures  at  the  conclusion  of  each  experimental  run. 
Between  each  experimental  run,  teams  completed  a  Raw  Task  Load  Index  (RTLX)  survey. 

Byers,  Bittner,  and  Hill  (1989)  proposed  the  RTLX  method  as  a  simplified  but  as-effective 
method  as  the  full  NASA-TLX.  The  RTLX  method  does  not  require  the  task-paired  comparison 
of  the  full  NASA-TLX  (Byers,  Bittner,  and  Hill,  1989). 

Communications 

Communications  were  categorized  in  three  basic  categories:  transfers,  requests,  and 
acknowledgements  (Entin  and  Entin,  2001).  Throughout  the  experiment,  both  video  and  audio  of 
the  participants  were  recorded.  These  recordings  were  analyzed  using  the  matrix  in  Figure  20. 
Explicit  communication  occurrences  were  aggregated  to  the  team  level.  The  data  were 
normalized  based  on  the  length  of  each  experimental  run.  Additionally,  communication  transfers 
were  divided  by  communication  requests  to  establish  an  “anticipation  ratio.”  Anticipation  ratios 
have  often  proved  more  useful  than  individual  rate  measures  for  understanding  team 
communications  (Entin  and  Entin,  2001).  Larger  anticipation  ratios  indicate  increased 
anticipation  of  team  member  information  needs.  Recall  that  the  most  effective  teams  seem  to 
share  their  mental  picture  of  the  situation  with  other  team  members  (Stout  et  al.,  1999).  These 
shared  mental  models  help  team  members  to  anticipate  the  needs  of  others;  and  this  permits  them 
to  either  provide  assistance,  as  it  is  required,  or  to  predict  and  pre-empt  the  need  for  assistance 
(Martin  and  Flin,  1997).  They  provide  team  members  with  a  common  understanding  of  who  is 
responsible  for  what  task  and  what  the  information  requirements  are  and  to  allow  team  members 
to  anticipate  one  another’s  needs  so  that  they  can  work  in  sync  and  adjust  their  behavior 
accordingly  (Smith-Jentsch,  Johnston,  and  Payne,  1998).  Hence,  the  presence  of  a  high 
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anticipation  ratio,  in  concert  with  other  indicators,  can  be  an  indicator  of  the  level  of  similarity  of 
a  team’s  shared  mental  model. 
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Figure  20.  Communications  matrix 


Results 

In  total,  72  runs  were  completed:  teams  completed  12  familiarization,  48  nominal,  and  12 
off-nominal  runs.  The  data  were  categorized  into  three  main  groups:  Performance, 
Communication,  and  Workload.  For  the  analyses  in  this  section,  the  following  statistical  tests 
were  used: 

•  General  Linear  Model  (GLM)  Analysis  of  Variance  (ANOVA) 

•  The  Mann- Whitney  U  Test 

•  Kruskal-Wallis  Test 

Statistical  Program  for  the  Social  Sciences  (SPSS)  13.0  for  Windows  Graduate  Student 
Version,  released  September  1,  2004,  was  used  to  perform  the  statistical  analyses.  Details 
concerning  the  computational  algorithms  can  be  found  at  http://www.spss.com. 
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The  GLM  multivariate  and  univariate  procedure  provides  regression  analysis  and  analysis  of 
variance  for  multiple  or  singular  dependent  variables  by  one  or  more  factor  variables  or 
covariates.  The  factor  variables  divide  the  population  into  groups.  This  procedure  is  commonly 
used  to  test  null  hypotheses  about  the  effects  of  factor  variables  on  the  means  of  various 
groupings  of  a  joint  distribution  of  dependent  variables  or  to  investigate  interactions  between 
factors,  as  well  as  the  effects  of  individual  factors  (SPSS  13.0  for  Windows  Graduate  Student 
Version).  Although  this  experiment  used  balanced  models,  both  balanced  and  unbalanced 
models  can  be  tested.  Post  hoc  Kolmogorov-Smirnov  tests  were  used  to  test  the  assumption  of 
normally  distributed  errors.  Additionally,  observed  power  was  calculated  with  each  ANOVA.  In 
these  analyses,  type  El  sums  of  squares  were  utilized. 

The  Mann-Whitney  U  test  was  used  as  the  primary  non-parametric  test  for  these  analyses.  It 
is  the  most  commonly  used  of  the  two-independent-samples  tests.  It  is  equivalent  to  the 
Wilcoxon  rank  sum  test  and  the  Kruskal-Wallis  test  for  two  groups.  Mann-Whitney  tests  that  two 
sampled  populations  are  equivalent  in  location.  The  observations  from  both  groups  are  combined 
and  ranked,  with  the  average  rank  assigned  in  the  case  of  ties.  The  number  of  times  a  score  from 
group  one  precedes  a  score  from  group  two,  and  the  number  of  times  a  score  from  group  two 
precedes  a  score  from  group  one,  are  calculated.  The  Mann-Whitney  U  statistic  is  the  smaller  of 
these  two  numbers.  The  Wilcoxon  rank  sum  W  statistic,  also  displayed,  is  the  rank  sum  of  the 
smaller  sample  (SPSS  13.0  for  Windows  Graduate  Student  Version). 

Performance 


The  measures  lane  violation  and  hard  decelerations  were  analyzed  using  a  GLM  ANOVA. 
No  significant  differences  were  found  (see  Table  1). 

The  measures  of  speed  violations,  total  navigation  errors  and  the  identification  of  speed 
zones  were  found  to  not  fit  the  normality  requirements  for  ANOVA.  Therefore,  each  was 
examined  using  a  Kruskal-Wallis  test  (see  Table  2).  They  were  also  found  to  have  non¬ 
significant  differences  across  information  levels.  However,  when  levels  I  and  H  (Navigator  had 
access  to  the  out-of-window  view)  were  combined,  and  levels  EE  and  IV  (Navigator  did  not  have 
access  to  the  out-of-window  view)  were  combined,  a  Mann-Whitney  test  was  conducted  and  a 
significant  effect  was  found.  Total  navigation  errors  significantly  decreased  during  experimental 
runs  when  navigators  did  not  have  access  to  the  out-of-the  window  view.  Table  3  contains 
statistics  associated  with  the  Mann-Whitney  Test  for  two  independent  samples. 

Performance  was  measured  for  the  off-nominal  condition  by  DT  from  presentation  of  roadblock 
to  action  taken  and  TT  to  get  back  on  the  route  toward  final  destination.  Significant  differences 
were  not  detected  when  the  information  levels  were  analyzed,  however,  when  they  were 
combined,  as  in  the  nominal  condition,  significant  effects  were  identified.  Information 
conditions  were  consolidated  to  reflect  whether  or  not  the  navigator  had  access  to  the  out-of-the 
window  view.  As  predicted,  the  first  roadblock  scenario  yielded  non-significant  results. 

Decision  Time  and  Total  Time  for  the  second  scenario  were  significant  (DT;  F  =  9.38,  p  =  .012 
and  TT;  F  =  5.29,  p  =  .044)  and  are  shown  in  Figure  21. 
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Table  1. 

ANOVA:  performance. 


Source 

Dependent  Variable 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Corrected  Model 

Hard  Decelerations 

85.167 

3 

28.389 

1.050 

.380 

Lane  Violations 

15.063 

3 

5.021 

.531 

.663 

Intercept 

Hard  Decelerations 

10443.000 

1 

10443.000 

386.182 

.000 

Lane  Violations 

14456.021 

1 

14456.021 

1529.309 

INFO 

Hard  Decelerations 

85.167 

3 

28.389 

1.050 

.380 

Lane  Violations 

15.063 

3 

5.021 

.531 

.663 

Error 

Hard  Decelerations 

1189.833 

44 

27.042 

Lane  Violations 

415.917 

44 

9.453 

Total 

Hard  Decelerations 

11718.000 

48 

Lane  Violations 

14887.000 

48 

Corrected  Total 

Hard  Decelerations 

1275.000 

47 

Lane  Violations 

430.979 

47 

Table  2. 

Kruskal-Wallis  test. 

Speed 

Violations 

Total 

Navigation 

Errors 

Speed  Zone 
Identification 

Chi-Square 

4.309 

5.787 

.899 

df 

3 

3 

3 

Asymp.  Sig. 

.230 

.122 

.826 

Group  Variable:  Level  of  Information  Distribution 


Table  3. 

Mann-Whitney  test. 


Navigator  had 
Access  to  Out-of- 
Window  View 

Navigation 

Errors 

Statistics 

Yes 

17 

Mann-Whitney  U 

204 

No 

4 

Wilcoxon  W 

504 

Z 

-2.167 

Sig. 

0.030 
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Communications 

The  communication  measures  were  analyzed  using  a  GLM  ANOVA.  The  results  of  the 
ANOVA  for  team  communication  measures  are  summarized  in  Table  4  (a=.  05).  Figure  22 
displays  the  overall  communication  rates  between  information  configurations.  While  requests 
for  information  showed  no  significant  change  across  information  distribution  levels,  there  were 
significant  differences  in  transfers  of  information,  acknowledgements  of  receipt  of  information, 
total  communications,  and  the  anticipation  ratio.  The  trend  was  a  general  increase  in  the  rate  of 
transfers  of  task  relevant  information,  acknowledgements,  and  total  communications  in  response 
to  a  complementary  configuration  of  information.  Additionally,  the  rate  of  transfers  of  non- 
relevant  information  decreased. 
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Task  Relevant  Requests 


Figure  22.  Nominal  communications  (automobile  domain). 
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Requests  were  recorded  when  either  team  member  explicitly  asked  for  information  or  action 
from  the  other  team  member.  Most  of  the  requests  were  for  information  regarding  the  team  task; 
specifically,  the  navigator  requested  numerous  actions  from  the  driver  (e.g.,  turn  right  on  E 
Street).  There  were  few  requests  for  non-relevant  information.  There  was  no  significant 
difference  in  the  team’s  requests  as  information  levels  varied. 

Transfers  were  categorized  in  the  same  fashion  as  requests.  Transfers  of  non-relevant 
information  decreased,  and  task  relevant  information  remained  constant  as  information  became 
more  complementary.  Teams  became  more  focused  on  the  navigation  task  and  the  transfer  of 
relevant  information  to  their  team  member.  Furthermore,  transfers  of  action  increased 
significantly.  An  action  transfer  was  recorded  when  one  team  member  verbalized  their  actions  to 
the  other  team  member.  The  majority  of  transfers  of  action  originated  from  the  driver.  During 
runs  where  information  was  complementary,  drivers  tended  to  verbalize  their  actions  more 
frequently. 

Acknowledgements  of  information  were  classified  as  general  or  specific  and  were  recorded 
when  a  team  member  verbally  acknowledged  that  they  received  information  from  the  other  team 
member.  A  general  acknowledgement  was  one  in  which  the  team  member  verbalized  that  they 
heard  the  information  (i.e.,  Roger,  Okay,  Got  it).  A  specific  acknowledgement  not  only 
confirmed  that  the  information  was  heard,  but  the  information  was  repeated  (i.e.,  Roger... turn 
right  on  E  Street).  This  type  of  acknowledgement  ensured  that  the  information  that  was  heard 
was  the  correct  information.  There  was  a  significant  increase  in  specific  and  total 
acknowledgements  as  information  became  more  complementary. 

Total  Communication  (TC)  was  a  combination  of  requests,  transfers,  and 
acknowledgements.  TC  significantly  increased  as  information  became  more  complementary. 

This  measure  indicated  that  the  team  verbalized  more  information,  but  indicated  nothing 
concerning  the  quality  of  information.  Based  on  the  increase  in  action  transfers  and 
acknowledgements,  TC  was  expected  to  increase. 

The  anticipation  ratio  was  calculated  using  transfers  and  requests  of  task  relevant 
information  only.  Using  the  communication  matrix  in  Figure  20,  the  anticipation  ratio  was 
calculated  by  dividing  all  task  relevant  information,  action,  and  acknowledgement  transfers  by  all 
task  relevant  information  and  action  requests  ([Transfers  of  Task  Relevant  Information  + 
Transfers  of  Action  +  Total  Acknowledgements]  /  Requests  for  Task  Relevant  Information). 

The  anticipation  ratio  increased  significantly  during  the  information  levels  where  the 
navigator  did  not  have  access  to  the  out-of-window  view.  This  indicated  that  during  these 
scenarios,  team  members  were  anticipating  the  information  required  by  the  other  team  member 
(Figure  23). 

Workload 


Workload  was  measured  using  the  RTLX.  Only  one  significant  difference  in  workload 
measurements  was  found  to  be  caused  by  levels  of  information  (Table  5);  the  navigators’ 
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Anticipation  Ratio 


perceived  effort  increased  in  the  complementary  configurations  when  compared  to  the  normal 
configurations.  Additionally,  further  analysis  showed  that  teams  experienced  a  higher  level  of 
frustration  during  the  off-nominal  runs  compared  to  the  nominal  runs. 
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Figure  23.  Anticipation  ratio  box 


Table  5. 

Individual  crewmember  workload  significance  levels. 
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Participant  effects 


For  the  first  experiment,  a  Greco-Latin  Square  design  was  used  to  minimize  any  effects  of 
the  scenario,  order,  and  training,  and  to  isolate  the  effects  of  the  independent  factors.  In  the 
second  experiment,  the  information  distribution  conditions  were  balanced  and  treated  as  a 
between-subjects  variable  for  this  run;  the  same  scenario  was  used  for  all  teams.  Participant 
effects  on  communication  and  performance  were  not  significant  to  the  findings  in  this 
experiment  and  are  detailed  in  Appendix  A. 

Conclusions 


All  hypotheses  were  supported  by  the  data  from  this  experiment.  The  results  of  this 
experiment  have  shown  that  as  information  is  distributed  in  a  complementary  condition: 

•  Team  performance  generally  remained  constant  under  nominal  conditions;  when 
complementary  levels  were  aggregated,  team  members  actually  navigated  more  accurately 
in  the  fully  complementary  configuration. 

•  Decision  time  decreased  during  off-nominal  conditions. 

•  Task  relevant  communications  increased. 

•  Non-relevant  information  transfers  decreased. 

•  Anticipation  ratio  (of  task  relevant  information)  increased. 

Designing  a  system  such  that  task  specific  information  is  provided  to  individual  team 
members  when  they  need  access  to  it,  while  limiting  their  access  to  non-relevant  information, 
changes  a  team’s  communication  strategy.  This  change  occurs  because  teams  develop 
complementary  team  mental  models  through  explicit  communication.  In  this  experiment,  the 
change  in  communications  was  manifested  through  a  significant  increase  in  explicit  feedback 
between  team  members  and  a  greater  anticipation  of  team  members’  information  requirements. 
These  are  both  essential  elements  in  the  development  of,  and  are  indicators  of  congruent  team 
shared  mental  models  (Cannon-Bowers,  Salas,  and  Converse,  1993;  Orasanu  and  Salas,  1993; 
Stout  et  al.,  1999). 

Current  literature  has  focused  on  training  as  an  effective  method  for  increasing  the  accuracy 
of  team  shared  mental  models  (Cannon-Bowers,  Salas,  and  Converse,  1993;  Duncan  et  ah,  1996; 
Minionis,  Zacarro,  and  Perez,  1995;  Orasanu  and  Salas,  1993).  Results  of  this  experiment 
indicate  that  it  is  possible  to  improve  teams’  mental  models  through  system  design  by  supporting 
the  development  of  complementary  team  mental  models.  Investigators  hypothesize  that  the 
results  of  this  experiment  are  not  domain  specific.  To  investigate  this  hypothesis,  a  similar 
experiment  was  conducted  in  a  separate  domain,  an  aircraft  cockpit.  Additionally,  more  explicit 
measures  of  the  complementariness  of  team  mental  models  and  individual  performance  were 
incorporated  in  this  second  experiment. 
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Experiment  II:  NUH-60  helicopter  simulator 


The  overarching  objective  of  this  study  was  to  test  the  hypothesis  that  providing  task 
relevant  information  to  individual  team  members  in  a  time  critical  environment,  while  limiting 
their  access  to  non-relevant  information,  would  foster  developing  complementary  team  mental 
models  with  corresponding  improvements  in  team  process  and  performance.  This  was 
accomplished  by  imposing  two  different  levels  of  complementariness  of  information  between 
team  members  and  observing  the  effect  on  team  communication,  team  and  individual  task 
performance,  team  and  individual  workload,  and  timeliness  and  effectiveness  of  team  decision 
making  in  nominal  and  off-nominal  conditions.  Variations  in  these  dependent  measures  provide 
insight  into  the  benefits  of  forming  complementary  mental  models  of  team  information 
requirements  and,  furthermore,  whether  the  accuracy  of  individual  mental  models  is  improved 
when  information  is  distributed  in  team  domain  based  on  task  requirements.  As  in  Experiment  I, 
a  condition  characterized  as  nominal  implies  that  no  experimental  changes/inputs  were  imposed 
on  the  team  which  altered  their  planned  navigational  flights.  During  off-nominal  conditions, 
teams  were  forced  to  alter  their  planned  flights  due  to  unexpected  events  imposed  by  the 
researchers.  Specific  hypotheses  that  support  this  overarching  objective  are: 

•  Hypothesis  1 :  Team  and  individual  performance,  during  nominal  conditions,  will  remain 
constant  when  information  is  complementary  when  compared  to  performance  during 
nominal  conditions  in  a  normal  cockpit  configuration. 

•  Hypothesis  2:  Team  and  individual  performance  during  off-nominal  conditions  will 
improve  when  information  is  complementary  when  compared  to  performance  during  off- 
nominal  conditions  in  a  normal  cockpit  configuration. 

•  Hypothesis  3:  Explicit,  task  relevant,  verbal  communications  will  increase  when 
information  is  complementary  in  both  nominal  and  off-nominal  conditions  when 
compared  to  communications  in  a  normal  cockpit  configuration. 

•  Hypothesis  4\  The  team’s  task  relevant  anticipation  ratio  will  increase  when  information 
distribution  is  complementary  when  compared  to  operations  in  a  normal  cockpit 
configuration. 

•  Hypothesis  5:  Team  member  information  requirement  rankings  will  be  more  similar  when 
information  is  complementary  when  compared  to  rankings  elicited  while  operating  in  a 
normal  cockpit  configuration. 

•  Hypothesis  6:  Individual  workload  ratings  will  remain  constant  between  information 
levels  regardless  of  cockpit  configuration. 

•  Hypothesis  7:  Team  member  ability  to  estimate  each  other’s  workload  rankings  will 
improve  when  information  is  complementary  when  compared  to  rankings  during 
operations  in  a  normal  cockpit  configuration. 

This  is  the  second  of  two  experiments  designed  to  investigate  the  effects  of  organizing 
information  within  a  complex  environment  to  enhance  the  development  of  Complementary  Team 
Mental  Models  (CMM);  the  first  experiment  was  conducted  in  an  automobile  simulator.  The 
complex  environment  selected  for  this  experiment  was  the  cockpit  of  a  Utility  Helicopter-60 
(UH-60),  a  modem  helicopter  that  is  currently  flown  in  both  the  Air  Force  and  the  Army.  The 
levels  of  organization  of  information  presented  to  the  participants  ranged  from  the  current 
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organization  of  information  in  the  cockpit  (normal)  to  a  complementary  organization  of 
information.  Some  information  was  available  to  both  team  members  (common),  while  other 
information  was  only  made  available  to  a  certain  team  member,  and  not  the  other,  to  support 
their  individual  and  team  tasks.  Various  metrics  were  used  to  examine  whether  reorganizing 
information  in  this  manner  had  a  positive  or  negative  effect  on  team  communication,  team  and 
individual  task  performance,  workload,  and  timeliness  and  effectiveness  of  team  decision¬ 
making. 

As  noted  in  the  previous  section,  analysis  of  the  experiment  conducted  in  an  automobile 
simulator  resulted  in  the  identification  of  significant  variations  in  some  of  the  measures  described 
above.  The  results  indicated  that  providing  task  relevant  information  to  individuals  while 
limiting  their  access  to  non-relevant  information  improved  the  effectiveness  of  communications 
and  led  to  better  performance  in  both  nominal  and  off-nominal  conditions.  These  results 
indicated  that  designing  a  system’s  information  distribution  in  a  manner  that  supports  the 
development  of  a  CMM  positively  effects  critical  team  performance  metrics. 

This  subsequent  experiment  (1)  investigated  the  design  implications  implied  by  the  results 
from  the  first  experiment  concerning  team  performance  through  more  in-depth  measures  of  team 
and  individual  performance,  (2)  investigated  the  effects  of  complementary  information 
distribution  on  individual  performance  by  adding  measures  for  individual  performance,  and  (3) 
tested  that  the  generalizability  of  the  results  found  in  the  automobile  experiment  were  not  domain 
specific.  Although  the  environment  of  the  first  experiment  was  clearly  complex,  participants 
were  familiar  with  the  task,  but  not  highly  trained  in  teamwork  or  navigation  techniques.  In 
contrast,  the  participants  in  the  second  experiment  were  professional  military  aviators,  trained  on 
team  resource  management  tasks  and,  specifically,  the  task  of  aircrew  navigation.  Through  the 
use  of  work  domain  analysis  and  task  analysis  techniques,  the  two  experiments  were  designed  to 
be  similar  in  task  requirements,  performance  measurements,  and  verbal/written  assessments. 

Method 

This  experiment  was  conducted  at  the  U.S.  Army  Aeromedical  Research  Laboratory 
(USAARL),  located  at  Fort  Rucker,  Alabama  using  military  helicopter  pilots  as  participants.  The 
main  parameter  was  the  complementariness  of  task  specific  information  available  to  team 
members.  During  the  experiment  data  were  collected  concerning  team  communications,  team 
workload,  information  requirements,  decision-making,  and  performance,  while  the  participants 
conducted  a  navigation  task  in  a  time  critical  situation.  Each  team  member  assumed  a  different 
role  in  the  team,  either  the  pilot  or  the  navigator  and  maintained  their  assigned  role  throughout 
the  entire  experiment  (i.e.,  there  was  no  role  switching). 

Participants 

Participants  were  20  U.S.  military  rated  aviators  tested  in  pairs  with  the  following 
characteristics: 

•  The  military  rank  ranged  from  Chief  Warrant  Officer  II  through  Lieutenant  Colonel. 
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•  Participants’  ages  ranged  from  24-57;  average  age  was  39  years. 

•  Total  flight  hours  ranged  from:  210  to  1 1,180;  average  was  3,290  hours. 

•  Each  team  was  required  to  have  at  least  one  team  member  rated  in  a  dual  engine  aircraft. 

•  50%  of  the  teams  were  qualified  in  the  UH-60. 

Participants  were  recruited  from  the  pilots  assigned  to  the  23d  Flying  Training  Squadron; 
USAARL;  and  1st  Battalion,  14th  Aviation  Regiment.  All  units  were  located  at  Fort  Rucker, 
Alabama.  Participants  received  flight  bags  donated  by  “Flight  Safety  International”  as  a 
memento  for  participation. 

Experiment  apparatus 

A  qualified  simulator  operator  operated  the  NUH-60  Black  Hawk  helicopter  flight  simulator 
used  for  this  study;  see  Figures  24  and  25.  This  provided  an  interactive  environment  in  which 
team  performance  could  be  observed,  while  certain  parameters  within  the  team  were  controlled. 
In  order  to  force  the  division  of  information  during  the  flight  segments,  the  view  of  the 
instruments  was  blocked  for  the  pilot  and/or  navigator.  This  was  accomplished  by  physically 
obstructing  the  view  of  certain  instruments  in  the  cockpit  with  cardboard  dividers  and  Velcro. 
Foggles  are  manufactured  glasses  that  are  used  as  a  tool  during  instrument  training  to  limit  the 
pilot's  field  of  vision.  This  assists  the  learning  process  by  encouraging  the  pilot  to  fly  with  only 
the  help  of  the  flight  instruments.  In  this  case,  they  were  used  to  restrict  the  navigators’  field  of 
view  and,  therefore,  information  available  to  him. 


Figure  24.  Cockpit  of  the  NUH-60  Black  Hawk  simulator. 
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Figure  25.  NUH-60  observer/operator  console. 


During  all  data  collection  runs,  video  and  sound  was  recorded.  The  cameras  were 
permanently  mounted  in  the  simulator.  The  recording  equipment  was  located  in  a  simulator 
monitoring  room.  The  flight  profiles  in  this  experiment  have  supporting  documentation  such  as 
maps,  route  cards,  and  approach  plates.  The  route  cards,  the  approach  plates,  and  airfield 
information  are  included  in  Appendix  B.  Aircraft  checklists  were  not  available  to  pilots; 
checklists  are  designed  as  a  back  up  to  information  that  is  required  to  be  memorized  and 
therefore,  should  not  have  a  negative  effect  on  the  experimental  outcome. 

Experiment  procedure 

•  Introductory  briefing. 

•  Sought  informed  consent  from  participants.  The  consent  form  contained  information 
regarding  the  scope  of  the  study,  the  tasks  that  were  performed  by  the  volunteers,  and  the 
volunteers’  rights  as  participants,  as  well  as  any  risks  associated  with  their  participation. 
This  form  is  contained  in  Appendix  B. 

•  Explained  the  experiment. 

•  Detailed  the  schedule  of  events. 

•  Oriented  participants  with  the  facilities. 

•  Collected  demographical  data  from  participants. 

Training  run 

A  training  run  was  conducted  to  ensure  that  each  participant  was  familiar  with  the  systems  in 
the  aircraft,  the  instrument  displays,  and  the  area  of  operation. 


Data  runs 


Participants  conducted  three  navigation  runs  during  which  data  were  collected,  two  nominal 
data  runs  (no  emergency  operations  required),  and  one  off-nominal  data  run  (emergency 
conditions  were  presented).  Immediately  following  each  of  the  navigation  runs  participants 
completed  surveys  regarding  their  workload,  performance,  etc.  for  themselves  and  for  their  team 
member. 

End  of  experiment  survey 

Following  the  complete  experiment,  individuals  completed  a  final  survey  concerning  overall 
themes  of  the  experiment.  The  entire  procedure  lasted  approximately  2.5  hours  for  each  team; 
this  included  introduction  briefing,  simulator  familiarization,  training  runs,  data  runs,  and  all 
surveys.  Participants  were  free  to  request  a  break  at  any  time. 

Experimental  tasks 

The  NUH-60  helicopter  flight  simulator  was  used  for  this  experiment.  The  experiment  was 
designed  to  collect  data  concerning  team  coordination,  decision-making  and  performance  while 
conducting  a  navigation  task  under  a  time  critical  situation.  Each  team  member  assumed  a 
different  role  in  the  team,  either  the  pilot  or  navigator.  The  team  was  given  a  map  and  textual 
directions  to  various  locations  within  the  simulated  environment.  They  were  asked  to  navigate  in 
two  specific  conditions  (nominal  and  off-nominal).  Under  nominal  conditions,  participants  were 
asked  to  conduct  two  navigation  runs.  During  these  runs,  the  pilot  and  navigator  were  placed  in 
scenarios  that  were  functionally  equivalent.  They  started  at  a  specified  location  and  were  asked 
to  navigate  to  a  destination;  the  information  provided  to  team  members  was  varied  during  these 
runs.  Teams  were  asked  to  maintain  control  of  the  aircraft,  obey  air  traffic  control,  and  navigate 
to  the  location  given  on  their  map  using  the  route  provided.  Teams  were  expected  to  conduct 
individual  tasks  (i.e.,  maintain  aircraft  control,  conduct  fuel  computations),  team  tasks  (i.e., 
navigation),  coordinate  their  actions  as  a  team,  and  communicate  their  actions,  intentions  and 
requests  for  information. 

They  started  at  a  specified  location  and  were  asked  to  depart  via  normal  take-off,  navigate  to 
a  destination,  and  land  at  the  destination  using  standard  route  cards;  the  independent  variable,  the 
level  of  information  provided  to  team  members,  varied  during  these  runs.  Teams  were  given  a 
map  with  a  route;  a  route  card  with  headings,  altitudes,  airspeeds,  and  checkpoints;  approach 
plates  for  local  airfields;  and  a  description  of  the  landing  area. 

The  off-nominal  run  consisted  of  one  experimental  run  similar  to  the  previous  two,  however, 
unexpectedly  to  the  team,  they  were  forced  to  enter  Instrument  Meteorological  Conditions 
(IMC);  subsequently  the  team  was  presented  an  in-flight  single  engine  alternator  failure.  This 
run  thus  assessed  team  responses  to  an  unexpected,  ‘off-nominal’  scenario.  In  addition  to  the 
expectations  for  the  nominal  condition,  teams  were  expected  to  formulate  a  plan  upon  entering 
IMC,  continue  to  navigate  to  their  final  destination,  and  properly  diagnose  the  in-flight 
emergency  procedure. 
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Experimental  design 


The  study  consisted  of  two  experiments  run  sequentially;  participants  were  unaware  that 
there  were  two  experiments.  The  first  experiment  design  consisted  of  two  runs  that  examined 
performance  under  nominal  conditions,  during  which  the  two  levels  of  information  distribution 
(complementary  and  normal)  were  varied.  This  experiment  was  balanced  within  subjects  to 
account  for  order  and  training  effects.  The  second  experiment  consisted  of  one  experimental  run 
similar  to  the  previous  two.  However,  the  team  was  required  to  deviate  from  normal  procedures 
(i.e.,  react  to  an  in-flight  emergency).  This  was  a  balanced  between  subjects  design  between  the 
two  information  distribution  levels. 

Independent  factors 

There  were  two  independent  factors  in  this  experiment:  complementariness  of  information 
and  operational  condition. 

Complementariness  of  information 

Two  levels  of  information  distribution  were  presented  to  the  participants:  normal  and 
complementary.  Team  member  information  requirements  were  based  on  the  domain  and  task 
analysis  described  in  the  section  on  the  determination  of  task  information  requirements.  This 
analysis  determined  the  information  that  each  team  member  required  access  to  in  order  to 
complete  their  individual  and  team  tasks.  The  fundamental  information  required  by  each  team 
member  is  shown  in  Figure  26.  In  the  normal  condition,  the  pilot  and  navigator  were  both  given 
identical  information;  i.e.,  they  both  had  access  to  all  information  displays  in  the  cockpit  and  they 
were  both  given  a  map  with  a  route  posted.  They  also  were  given  a  route  card  with  headings, 
altitudes,  airspeeds,  and  checkpoints;  approach  plates  for  local  airfields;  and  a  description  of  the 
landing  area.  Under  the  complementary  condition,  individual  team  members  were  only  provided 
access  to  information  relevant  to  their  individual  tasks  and  for  their  defined  roles  in  team  tasks. 
Specifically,  only  the  navigator  was  given  a  map  with  a  route  posted;  a  route  card  with  headings, 
altitudes,  airspeeds,  and  checkpoints;  approach  plates  for  local  airfields;  and  a  description  of  the 
landing  area.  Likewise,  the  navigator  wore  Foggles  preventing  out-of-windscreen  viewing.  The 
pilot  had  access  to  all  flight  instruments,  but  access  to  engine  related  performance  instruments 
was  restricted  to  the  navigator.  The  navigator  was  not  allowed  to  visually  share  the  map,  route 
card,  etc.  with  the  pilot. 
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Figure  26.  Information  sources. 


Operational  condition 

Two  operational  conditions  were  presented  to  the  participants:  nominal  and  off-nominal. 
During  nominal  conditions  teams  maintained  visual  flight  rules  throughout  the  simulation,  and 
they  experienced  no  system  malfunctions  during  the  mission.  During  off-nominal  conditions, 
teams  experienced  inadvertent  instrument  metrological  conditions  (DMC)  and  a  single  engine 
alternator  failure  during  the  flight. 

The  flight  profile  incorporated  various  phases  of  flight  during  visual  meteorological 
conditions  (VMC)  and  instrument  meteorological  conditions  (IMC).  The  profile  has  three 
sections  to  be  flown  in  order,  each  lasting  approximately  15  minutes.  Flight  phases  of  interest 
during  VMC  flight  include  take-off,  VMC  flight  in  cruise  (above  200  ft  AGL),  and  landing. 

Flight  phases  of  interest  during  IMC  flight  include  take-off,  straight  and  level  flight,  climbs, 
descents,  standard  rate  turns,  and  landing.  All  flight  maneuvers  were  flown  in  accordance  with 
Army  standards.  A  plan  view  of  the  profile  is  included  in  Appendix  B  and  the  profile  segments 
are  described  in  Table  6. 
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Table  6. 
Flight  profiles. 


TASK 

TASK  DESCRIPTION 

BHBi 

■1119 

1 

■ 

VMC  Takeoff  (500fpm) 

230 

0  AGL  -  900 
MSL 

0-100 

2 

Straight  and  Level  -  Cruise 

To  Waypoint  1 

226 

900  MSL 

100 

3 

Straight  and  Level  -  Cruise 

To  Waypoint  2 

239 

1300  MSL 

110 

II 

Straight  and  Level  -  Cruise 

To  Waypoint  3 

240 

1100  MSL 

90 

5 

VMC  Approach 

240 

1100 MSL  -  0 
AGL 

90-0 

6 

VMC  Takeoff  (500fpm) 

240  -  332 

0 AGL -1000 
MSL 

0-80 

■ 

Straight  and  Level  -  Cruise 

To  Waypoint  4 

332 

1000  MSL 

80 

8 

Straight  and  Level  -  Cruise 

To  Waypoint  5 

322 

1400  MSL 

90 

9 

Straight  and  Level  -  Cruise 

To  Waypoint  6 

348 

1100  MSL  -  0 
AGL 

80 

10 

VMC  Approach 

348 

1100  MSL-0 
AGL 

80-0 

11 

VMC  Takeoff  (500fpm) 

348  -  085 

0  AGL -1000 
MSL 

0-120 

12 

Straight  and  Level  -  Cruise 

To  Waypoint  7 

085 

1000  MSL 

120 

13 

| 

085  -  070 

1000  MSL- 
2000  MSL 

120 

14 

Straight  and  Level  (Vectors) 

070 

2000  MSL 

120 

15 

Straight  Climb 

070 

2000  MSL  - 
4000  MSL 

120 

14 

Straight  and  Level/Emergency 
Procedure  (Alternator  Failure) 

070 

4000  MSL 

120 

15 

Straight  Descent 

070 

4000  MSL- 
2100  MSL 

120 

16 

Straight  and  Level 

070 

2100  MSL 

120 

17 

Right  Standard  Rate  Turn 

070-  140 

2100  MSL 

120 

18 

ILS  Rwy  23  (Campbell  AAF) 

225 

2100  MSL -773 
MSL 

120 

TIME 

(MIN+SEC) 

-2+00 

-6+18 

-3+51 

-3+30 

-2+00 

-2+00 

-3+20 

-5+30 

-4+12 

-2+00 

-2+00 

2+00 

2+00 

1+00 

2+00 

2+00 

4+00 

1+00 

0+30 

3+00 
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Dependent  factors 

During  the  experiment,  data  were  collected  through  several  means.  The  simulator  compiled 
performance  data  (root  mean  square  error  [RMSE]  for  altitude,  airspeed,  rate  of  climb,  rate  of 
descent,  heading,  and  rate  of  turn).  The  screen  of  the  out-the-window  view  was  synchronized 
with  video  of  the  pilot  and  navigator  via  split  screen  video,  and  an  observer  was  present  behind 
the  cockpit,  in  an  observation  area,  to  record  any  anomalies.  Furthermore,  immediately 
following  each  of  the  scenarios,  participants  were  asked  to  complete  surveys  regarding  their 
workload,  performance,  information  requirements,  etc.  Following  the  complete  experiment, 
individuals  were  asked  to  complete  a  final  survey  concerning  overall  themes  of  the  experiment. 
The  data  were  categorized  into  three  main  groups:  Performance,  Communication,  and  Survey 
(RTLX,  information  requirements,  and  demographics). 

Performance 

During  nominal  conditions,  performance  was  gauged  by  flight  performance  measures 
recorded  by  the  simulator:  RMSE  of  airspeed,  altitude,  and  heading,  and  rate  of  climb. 
Additional  task  performance  measures  were  evaluated: 

•  Completion  of  required  radio  calls :  Teams  were  given  a  list  of  radio  calls  required  in 
each  flight  leg.  This  metric  is  represented  by  a  percentage  of  those  calls  that  were 
actually  completed. 

•  Calculation  of  estimated  time  enroute:  During  each  run,  navigators  were  required  to 
calculate  the  estimated  time  enroute  for  two  legs  of  the  route. 

•  Initiation  of  a  fuel  consumption  check :  Teams  were  required  to  initiate  a  fuel 
consumption  check  during  each  run;  this  metric  indicates  whether  or  not  this  was 
completed. 

•  Navigation  and  process  errors :  Navigation  errors  concerned  time,  heading,  distance, 
altitude,  etc.  Process  errors  include  using  the  wrong  frequencies  and  using  the  wrong 
procedures. 

During  the  off-nominal  condition,  in  addition  to  the  measures  used  in  the  nominal  condition, 
the  following  task  performance  measures  were  assessed: 

•  1IMC  call  time  to  Campbell  Army  Airfield  ( CAAF):  This  was  the  time  recorded  from 
when  the  team  entered  EMC  until  they  notified  CAAF. 

•  Proper  IIMC  procedures'.  An  indicator  of  whether  the  team  performed  the  proper  IIMC 
procedures  in  accordance  with  the  aircrew-training  manual. 

•  Diagnosis  time  of  emergency.  Time  was  recorded  from  the  presentation  of  the  emergency 
until  the  team  verbalized  the  problem  or  the  corrective  action  needed. 

•  Diagnosis  of  the  proper  emergency  procedure :  This  metric  indicated  whether  the  proper 
emergency  procedure  was  executed. 

•  Emergency  call  time  to  CAAF:  This  was  the  time  recorded  from  when  the  team  was 
presented  with  the  emergency  until  they  notified  CAAF. 
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Communication 


Communication  measures  were  determined  using  the  matrix  in  Figure  27.  Verbal 
communications  were  categorized  in  three  basic  categories:  transfers,  requests,  and 
acknowledgements  (Entin  and  Entin,  2001).  Throughout  the  experiment,  both  video  and  audio  of 
the  participants  were  recorded.  Following  the  experiment,  an  analyst  reviewed  the  videos  of  each 
team  and  recorded  the  data  in  this  format.  The  communication  data  were  normalized  based  on 
the  length  of  each  experimental  run.  Additionally,  communication  transfers  were  divided  by 
communication  requests  to  establish  an  “anticipation  ratio.”  Anticipation  ratios  have  often 
proved  more  useful  than  individual  rate  measures  for  understanding  team  communications  (Entin 
and  Entin,  2001).  The  first  study  conducted  in  this  research  corroborated  the  finding  in  literature 
that  larger  anticipation  ratios  indicate  increased  anticipation  of  team  member  information  needs. 
Variations  in  team  communication  measures  provide  insight  into  the  complementariness  of  the 
teams’  mental  models  of  team  interaction  and  the  accuracy  of  individual  mental  models  of  the 
environment. 


T ype  &  Content 

N  aviqator  to  P  ilot 

Pilot  to  Navigator 

Total 

Request 

Task  R  elevant 
Inform  ation 

Non  Task  Relevant 
Information 

A  ction 

T  ransfers 

Task  R  elevant 
Information 

Non  Task  Relevant 
Information 

Perform  ing/  W  ill 
Perform  Action 

Acknowledgements  of 
Info  Receipt 

General  (okay.roger) 

Specific  (roger... right  on 
turn  to  180  degrees) 

Figure  27.  Communications  matrix  (NUH-60). 
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Workload 


Workload  was  measured  through  the  use  of  the  NASA  TLX.  Between  each  experimental 
run  teams  completed  a  RTLX  survey.  Six  subscales  of  workload  were  collected:  mental  demand, 
physical  demand,  temporal  demand,  performance,  effort,  and  frustration.  Byers,  Bittner,  and  Hill 
(1989)  proposed  the  RTLX  method  as  a  simplified  but  as  effective  method  as  the  full  NASA 
TLX.  The  RTLX  method  does  not  require  task-paired  comparison  as  the  full  NASA  TLX  does 
(Byers,  Bittner,  and  Hill,  1989).  These  measures  were  collected  for  self  assessment  and 
assessment  of  team  members’  workload.  Appendix  B  contains  the  questions  and  rating  scale  for 
this  instrument. 

Information  requirements 

After  each  scenario,  participants  were  asked  to  rank  the  importance  of  their  information  sources 
during  each  phase  of  flight  (Figure  28);  participants  ranked  the  importance  of  the  types  of 
information  during  different  phases  of  flight.  They  rated  how  important  each  type  of  information 
was  to  them  and  how  important  they  thought  it  was  for  their  team  member.  This  was  done  for 
each  phase  of  flight:  take-off,  enroute  navigation,  and  landing  during  nominal  conditions  and 
upon  entering  IMC  and  dealing  with  the  emergency  procedure  in  off-nominal  conditions.  This 
survey  is  included  in  Appendix  B. 


Phase  of  Flight 

Take-off 

En-route 

During  IIMC 
procedure 

During  Emergency 
procedure 

Landing 

Information 

Type 

Description 

■ 

■ 

■ 

■ 

Current  Flight 
Information 

The  current  airspeed,  altitude, 
heading,  climb  rate  etc.  of  the 
aircraft 

■ 

■ 

■ 

■ 

■ 

Future  Flight 
Information 

The  future  airspeed,  altitude, 
heading,  climb  rate  etc.  that 
you  will  be  required  to  fly 

■ 

■ 

■ 

■ 

1 

Current 

Environmental 

Information 

Description  of  the 
environment,  terrain, 
checkpoints,  etc.  that  you  are 
currently  flying  in/over 

1 

1 

1 

1 

1 

Future 

Environmental 

Information 

Description  of  the 
environment,  terrain, 
checkpoints,  etc.  that  you  will 
oe  flying  in/over  in  the  near 
future 

■ 

■ 

■ 

■ 

■ 

Aircraft 

Instrumentation 

Status 

Status  of  the  aircraft 
nstrumentation  (fuel,  engine 
nstruments,  etc) 

Figure  28.  Information  requirements 


Scenario  design 


A  total  of  four  scenarios  were  developed:  one  training  scenario,  two  nominal  scenarios,  and 
one  off-nominal  scenario.  Following  the  introduction,  individuals  were  taken  through  the 
training  session.  The  training  was  designed  to  familiarize  the  pilot  with  the  feel  of  the  simulator 
controls,  the  simulated  area  of  operation,  and  the  type  of  scenarios  that  they  would  be  expected  to 
complete.  No  data  collection  was  done  at  this  time.  The  simulator  operator/instructor  pilot 
described  the  instrumentation  in  the  cockpit;  then  the  team  conducted  a  normal  take-off  from 
CAAF  and  normal  landing  to  a  forward  arming  and  refueling  point  (FARP).  All  teams  generally 
used  the  same  training  scenario.  Pilots  that  were  qualified  in  the  UH-60  required  less 
familiarization  of  the  cockpit  instruments  than  pilots  that  were  not  qualified  in  the  UH-60. 

Two  nominal  scenarios  were  next  presented  to  the  participants.  The  nominal  runs  were 
always  during  the  first  two  legs  of  the  route  structure  (Figure  29).  During  these  runs,  the  teams 
faced  similar  scenarios  for  each  leg.  These  scenarios  created  time  pressure  and  emphasized  the 
importance  of  mission  success.  During  each  scenario,  teams  started  at  a  specified  location  and 
were  asked  to  depart  via  normal  take-off,  navigate  to  a  destination,  and  land  at  the  destination 
using  standard  route  cards.  The  independent  variable,  complementariness  of  information 
provided  to  team  members,  was  varied  during  these  runs.  This  investigation  was  not  intended  to 
test  the  effects  of  pre-planning;  therefore,  teams  were  not  given  a  dedicated  time  to  plan  a 
teamwork  strategy.  They  were  given  a  map  with  a  route  posted;  they  were  also  given  a  route  card 
with  headings,  altitudes,  airspeeds,  and  checkpoints,  approach  plates  for  local  airfields,  and  a 
description  of  the  landing  area.  The  runs  were  each  approximately  1 5  minutes  long  and  a  break 
was  given  between  them. 


Figure  29.  Route  structure. 


The  one  off-nominal  scenario  was  always  presented  during  the  third  leg  of  the  route 
structure.  This  run,  unexpectedly  to  the  team,  caused  them  to  enter  IIMC;  subsequently  the  team 
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was  presented  an  in-flight  aircraft  emergency,  a  single  engine  alternator  failure,  along  their 
designated  route.  This  run  was  designed  to  assess  team  responses  to  an  unexpected,  ‘off- 
nominal’  scenario.  The  two  levels  of  information  distribution  were  treated  as  a  between-subjects 
variable;  the  same  scenario  was  used  for  all  teams. 

The  nominal  scenarios  were  functionally  equivalent;  off-nominal  scenarios  were  identical. 

A  plan  view  of  the  three  data  collection  runs,  a  description  of  scenarios  and  route  cards,  and  a 
description  of  the  airfield  and  approaches  available  at  the  airfield  are  included  in  Appendix  B. 
Examples  of  a  scenario  and  route  card  (Figure  30),  for  nominal  conditions,  are  provided  below. 
Example  nominal  condition  scenario 


Waypoint 

WPT  Description/ 
Coordinates 

Course 

ETE 

Maneuver  Description 

Distance 

(nm/km) 

VMC  takeoff  230°  from 

Campbell  Army  Airfield 

1 

Road  Intersection 
N36°32.91’ 
W087°39.93’ 

226° 

TBD 

Cruise  Flight 

900’  MSL,  100  KIAS 

10.5/19.3 

2 

Mouth  of  Inlet  (Bridge) 
N36°28.52’ 
W087°44.ir 

239° 

TBD 

Cruise  Flight 

1300’  MSL,  110  KIAS 

7.1/13.2 

3 

FARP  1 
N36°25.00’ 
W087°49.99’ 

240° 

TBD 

Cruise  Flight 

1100’  MSL,  90  KIAS 

5.8/10.8 

Figure  30.  Sample  route  card. 


This  was  given  and  read  to  the  pilot  and  navigator: 

You  are  assigned  the  mission  of  transporting  two  passengers  (captured  A1  Qaeda  operatives) 
from  Campbell  Army  Airfield  to  FARP  1.  The  mission  is  critical  and  must  be  completed  to 
avoid  an  international  incident.  The  only  reason  to  abort  or  modify  the  mission  will  be  in  the 
event  of  an  emergency  situation  requiring  a  landing  as  soon  as  possible.  Due  to  security 
concerns,  a  special  VMC  flight  corridor  to  the  FARP  has  been  established.  The  GPS  is 
inoperative  and  you  must  navigate  by  pilotage  and  dead  reckoning.  You  must  follow 
altitude,  airspeed  and  ground  track  restrictions  in  accordance  with  the  waypoint  card  in  order 
to  comply  with  the  corridor  requirements.  Navigator  is  required  to  calculate  estimated  time 
enroute  for  each  checkpoint.  In  addition  to  standard  ATC  radio  calls,  the  pilot  is  required  to 
contact  ATC  no  more  that  one  minute  prior  to  crossing  each  checkpoint  final  destination. 
Maintain  visual  flight  rules  (VFR).  Weather  for  the  mission  is  winds  calm,  1500  feet 
overcast;  3  statute  miles  visibility  (ETA  through  1  hour).  Use  ATC  frequencies  per  the 
DOD  FLIP.  FARP  1  frequency  is  34.15. 
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Results 


In  total,  40  runs  were  flown:  10  teams  completed  10  familiarization,  20  nominal,  and  10  off- 
nominal  runs.  The  data  were  categorized  into  three  main  groups:  Performance,  Communication, 
and  Survey  (NASA  RTLX,  information  requirements,  and  demographics). 

The  following  statistical  tests  were  used: 

•  GLM  ANOVA 

•  The  Mann- Whitney  U  test 

•  Kruskal-Wallis  test 

•  Spearman  Rank  Order  Correlation  Coefficient 

•  Binomial  test 

SPSS  13.0  for  Windows  Graduate  Student  Version,  released  September  1,  2004,  was  used  to 
perform  the  statistical  analyses.  Details  concerning  the  computational  algorithms  can  be  found  at 
http://www.spss.com. 

Performance 


Measures  of  flight  performance  were  analyzed  using  a  GLM  ANOVA.  Tasks  1-10  (Table  6) 
were  completed  during  nominal  conditions.  The  first  five  tasks  were  completed  during  the  first 
run;  the  second  five  tasks  were  completed  during  the  second.  During  nominal  conditions,  no 
significant  differences  were  found  to  exist  due  to  changes  in  the  distribution  of  information 
within  the  cockpit. 

The  measures  of  task  performance  did  not  fit  the  normality  requirements  for  ANOVA. 
Therefore,  each  was  examined  using  a  Mann-Whitney  test  to  identify  the  main  effects  of  the 
independent  variable  (Table  7).  Total  errors  (p=  .015)  were  significant;  both  process  errors  and 
navigation  errors  were  on  the  borderline  significance  at  the  5%  level.  Table  8  indicates  the  total 
count  of  errors  committed  by  category.  Fewer  errors  were  committed  in  the  nominal  condition 
when  the  information  available  was  distributed  in  a  complementary  manner. 
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Table  7. 

Mann-Whitney  non-par  test:  subjective  performance  measures  (nominal). 


Fuel 

ETE 

Navigation 

errors 

Process 

Errors 

Total 

errors 

Mann-Whitney  U 

30.000 

45.000 

34.500 

26.000 

35.000 

20.000 

Wilcoxon  W 

85.000 

100.000 

89.500 

81.000 

90.000 

75.000 

Z 

-1.744 

-.457 

-1.329 

-1.934 

-1.831 

-2.437 

Asymp.  Sig. 

.081 

.648 

.184 

.053 

.067 

.015 

Table  8. 

Errors  committed  (nominal). 


Complementary 

Information 

Navigation 

Errors 

Process 

Errors 

Total 

Errors 

No 

12 

3 

15 

Yes 

5 

0 

5 

Tasks  11-20  from  the  flight  profile  table  (Table  6)  were  performed  during  the  off-nominal 
run.  Once  again,  the  flight  performance  measures  were  analyzed  using  a  GLM  ANOVA.  The 
results  are  shown  in  Table  9.  During  task  11,  entry  into  EMC,  the  RMSEfor  airspeed  was  found 
to  be  significantly  different  as  the  independent  variable  changed  (p=  .020).  The  box  plot  of  the 
root  mean  square  error  of  airspeed  shows  that  the  median  and  mean  RMSE  improved  in  the 
complementary  condition  from  23.5  to  19.5  and  22.25  to  19.08,  respectively  (Figure  31). 
Furthermore,  the  box  plot  indicates  that  the  interquartile  range  of  error  in  the  complementary 
condition  is  less  than  the  normal  condition. 

A  number  of  the  measures  of  task  performance  could  be  analyzed  using  the  GLM  ANOVA; 
however,  others  were  found  not  to  fit  the  normality  requirements  for  ANOVA.  The  non- 
parametric  measures  were  examined  using  a  Mann-Whitney  test  to  identify  the  main  effects  of 
the  independent  variable.  Significant  results  were  identified  when  examining  the  percent  of  radio 
calls  completed  and  emergency  procedure  diagnosis  time.  These  are  discussed  below. 
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Table  9. 

ANOVA:  objective  performance  measures:  airspeed  (off  nominal) 


Dependent  Variable:  Asp^rmse 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

KB 

Noncent. 

Parameter 

Corrected  Model 

25.091 

25.091 

8.352 

8.352 

Intercept 

4270.836 

mmm 

4270.836 

1421.581 

1421.581 

Compart 

25.091 

25.091 

8.352 

8.352 

Error 

24.034 

3.004 

Total 

4319.960 

|^k| 

■ 

Corrected  Total 

49.125 

■ 

a  Computed  using  alpha  =  .05 


Across  the  complete  flight  profile  for  off  nominal  conditions  the  completion  of  required 
radio  calls  increased  when  the  team  was  provided  with  complementary  information.  Table  10 
tabulates  the  results  of  the  Mann-Whitney  test  for  the  percent  of  radio  calls  completed.  The  p- 
value  is  .053,  slightly  above  the  alpha  level  used  for  this  analysis  (.05),  but  there  is  merit  in 
presenting  the  findings.  Figure  32  shows  the  direction  of  the  difference  and  indicates  the 
difference  in  completed  radio  calls  under  the  normal  condition;  the  navigators  completed  one 
hundred  percent  of  the  radio  calls  required  during  the  complementary  condition.  Both  the 
median  and  mean  percent  of  completed  calls  increased  across  conditions,  and  performance  was 
more  consistent  during  the  complementary  condition. 

Table  10. 

Mann-Whitney  test  for  percent  of  radio  calls  completed. 


calls  percent 

Mann-Whitney  U 

5.000 

Wilcoxon  W 

20.000 

2 

-1.936 

Asymp.  Sig.  (2-tailed) 

.053 

Figure  32.  Box  plot  for  percent  of  required  radio  calls  completed. 
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Additionally,  the  diagnosis  time  of  aircraft  emergency ,  measured  in  seconds,  was  analyzed 
using  a  GLM  ANOVA  and  was  found  to  have  significant  differences  between  the  levels  of 
information  complementariness;  the  p-value  was  .007  with  an  observed  power  of  .912.  Complete 
results  of  the  ANOVA  are  shown  in  Table  11.  Figure  33  illustrates  the  direction  of  the  difference 
and  highlights  the  significant  decrease  in  diagnosis  time.  Both  the  median  and  mean  diagnosis 
time  decreased  across  conditions  from  130  to  49  and  1 15.3  to  39.4  seconds,  respectively. 
Furthermore,  the  standard  deviation  decreased  from  approximately  34  to  20  seconds. 

Table  11. 

Diagnosis  time  of  aircraft  emergency  ANOVA. 


Communications 


For  each  run,  communications  were  aggregated  and  analyzed  at  the  team  level.  Therefore, 
each  team  contributed  two  sets  of  data  (runs  one  and  two)  under  nominal  flight  conditions  and 
one  set  of  data  (run  three)  when  exposed  to  off-nominal  flight  conditions. 

A  GLM  ANOVA  was  used  to  evaluate  all  communication  rates  in  the  nominal  condition. 
The  results  are  presented  in  Table  12  and  the  significant  dependent  variable  means  are  displayed 
in  Figure  34.  The  following  intermediate  categories  of  team  communication  rates  increased 
significantly  when  the  team  was  exposed  to  a  complementary  information  distribution:  Team 
Transfers  of  Action  (RTRA),  Team  Transfers  of  Task  Relevant  Information  (RTTTRI),  Team 
Total  Transfers  (RTTT),  and  Team  Total  Communications  (RTTC). 

Table  12. 

Nominal  communications  ANOVA. 


Source 

Dependent  Variable 

Type  III  Sum 
of  Squares 

df 

MSE 

LL 

Siq. 

Observed 

Power3 

Compart 

RTRTRI 

.034 

1 

.034 

.163 

RTRIT 

.034 

1 

.034 

IHl 

.163 

RTRA 

7.718 

1 

7.718 

'IB 

.000 

.987 

RTTTRI 

15.671 

1 

15.671 

■SSf 

.002 

.920 

RTTNTRI 

.159 

1 

.159 

MB 

.005 

.861 

RTTA 

.171 

1 

.171 

1.364 

.258 

.198 

RTTT 

15.785 

1 

15.785 

14.840 

.001 

.953 

RTAG 

.011 

1 

.011 

.057 

.814 

.056 

RTAS 

.544 

1 

.544 

3.810 

.067 

.455 

RTAT 

.707 

1 

.707 

1.744 

.203 

.240 

RTTC 

60.470 

1 

60.470 

24.508 

.000 

.997 

RTAR 

106.320 

1 

106.320 

9.818 

.006 

.842 

a-  Computed  using  alpha  =  .05 
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Figure  34.  Communication  means  bar  graph  (nominal). 


Furthermore,  the  rate  of  Transfer  of  Non-Task  Relevant  Information  (RTTNTRI)  decreased 
in  the  complementary  configuration,  as  did  the  Team  Anticipation  Ratio  (RTAR).  Each  detected 
change  was  accompanied  by  a  strong  observed  power  calculation;  the  lowest  observed  power  was 
.842. 


A  GLM  ANOVA  was  also  used  to  evaluate  all  communication  rates  in  the  off-nominal 
condition.  The  results  are  presented  in  Table  13  and  the  significant  dependent  variable  means  are 
displayed  in  Figure  35.  The  results  in  the  off-nominal  condition  were  very  similar  to  the  nominal 
condition.  The  following  team  communication  rates  increased  significantly  when  the  team  was 
exposed  to  a  complementary  information  distribution:  RTRA,  RTTTRI,  RTTT,  Team 
Acknowledgements  Specific  (RTAS),  and  RTTC.  In  addition,  the  RTAR  decreased  in  the 
complementary  configuration.  Each  detected  change  was  accompanied  by  a  strong  observed 
power  calculation;  the  lowest  observed  power  was  .657. 


Table  13. 

Off-nominal  communications  ANOVA. 


Source 

Dependent  Variable 

Type  III  Sum 
of  Squares 

df 

MSE 

F 

Sip. 

Observed 

Power 

Compart 

RTRTRI 

.035 

1 

.035 

2.317 

.166 

.269 

RTRNTRI 

.006 

1 

.006 

1.000 

.347 

.143 

RTRIT 

.012 

1 

.012 

.869 

.378 

.131 

RTRA 

6.074 

1 

6.074 

22.520 

.001 

.985 

RTTTRI 

9.493 

1 

9.493 

12.348 

.008 

.867 

RTTNTRI 

.004 

1 

.004 

.699 

.427 

.115 

RTTA 

.041 

1 

.041 

.871 

.378 

.131 

RTTT 

10.383 

1 

10.383 

20.508 

.002 

.976 

RTAG 

.001 

1 

.001 

.021 

.888 

.052 

RTAS 

.116 

1 

.116 

7.261 

.027 

.657 

RTAT 

.094 

1 

.094 

1.271 

.292 

.169 

RTTC 

37.248 

i 

37.248 

37.525 

.000 

1.000 

RTAR 

15.712 

1 

15.712 

7.903 

.023 

.693 

Normal 
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Figure  35.  Communication  means  bar  graph  (off-nominal). 
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The  disparity  of  results  concerning  the  task  relevant  anticipation  ratio  between  experiment 
one  and  two  was  radical.  ANOVA  of  the  communication  metrics  for  this  experiment  indicate  a 
significant  p-value  for  both  the  nominal  and  off-nominal  conditions;  p  =  .006  and  p  =  .023, 
respectively;  across  these  conditions,  the  anticipation  ratio  decreased  in  the  complementary 
configuration.  The  significant  decrease  in  the  median  and  mean  between  the  two  levels  of 
information  distribution  has  been  tabulated  in  Table  14  and  illustrated  in  Figure  36.  In  addition, 
each  measure  of  dispersion  in  Table  15  shows  a  decrease,  as  well,  from  the  normal  configuration 
to  the  complementary  configuration,  to  include  interquartile  range,  standard  deviation,  variance, 
and  range  (min/max). 


Table  14. 

Descriptive  statistics  for  anticipation  ratio. 


Information  Configuration 

Normal 

Complementary 

Mean 

7.02 

3.11 

Median 

5.76 

3.19 

Variance 

16.35 

0.92 

Std.  Deviation 

4.04 

0.96 

Minimum 

1.94 

1.71 

Maximum 

18.25 

5.86 

Range 

16.31 

4.15 

Interquartile 

Range 

3.75 

0.96 

Figure  36.  Task  relevant  anticipation  ratio  box  plot. 
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Information  requirements 


This  section  describes  the  results  of  analysis  of  a  selection  of  survey  questions  concerning 
team  member  information  requirements.  Specifically,  the  following  results  address  the  ability, 
and  difficulty,  of  each  team  member  to  identify  and  anticipate  information  required  by  the  other 
team  member  in  the  normal  and  complementary  cockpit  configurations.  Data  were  collected 
under  nominal  and  off-nominal  conditions. 

The  information  requirements  matrix  (Figure  28)  was  used  to  record  team  member  rankings. 
Three  phases  of  flight  were  addressed  in  the  nominal  condition:  takeoff,  enroute,  and  landing. 

The  three  nominal  phases,  plus  two  additional  phases,  were  addressed  during  the  off-nominal 
condition:  13MC  procedures  and  emergency  procedures.  Rankings  of  information  requirements 
from  the  pilot  and  navigator  were  matched  by  phase  of  flight  and  a  correlation  matrix  was 
developed  using  the  Spearman  Rank  Order  Correlation  Coefficient.  Relevant  correlations  were 
selected  from  the  matrix  and  were  analyzed  using  a  GLM  ANOVA  for  significant  differences. 
The  ANOVA  performed  for  the  nominal  condition  found  no  significant  differences.  On  the 
contrary,  differences  in  the  means  of  correlation  coefficients  were  significant  in  the  off-nominal 
condition  due  to  changes  in  the  information  distribution  of  the  cockpit;  the  full  ANOVA  is 
displayed  in  Table  15.  Figure  37  illustrates  the  increase  in  the  median  and  mean  from  the  normal 
distribution  to  a  complementary  distribution  of  cockpit  information.  The  inner  quartile  range  is 
consistent  between  information  levels  as  is  the  standard  deviation  and  variance. 

The  following  questions  were  asked  of  each  participant  at  the  end  of  the  experiment: 

•  In  which  condition  were  you  more  aware  of  your  team  member’s  information 
requirements  (normal  or  complementary)? 

•  In  which  condition  were  your  own  information  needs  more  clearly  (normal  or 
complementary)  ? 

The  results  to  these  questions  are  displayed  in  Table  16.  Over  80  percent  of  the  participants  that 
answered  the  survey  question  indicated  that  the  complementary  distribution  of  information 
within  the  cockpit  provided  more  clarity  to  their  own  information  requirements,  and  they  felt  they 
understood  their  team  members’  information  requirements  better.  A  binomial  test  was  used  to 
analyze  the  responses;  the  test  proportion  used  for  the  analysis  was  .5  (50%),  and  both  answers 
were  significant  at  the  5%  level. 

Workload 


Workload  across  information  configurations  was  analyzed  in  three  different  manners:  (1) 
Individual  workload  ratings  were  analyzed  to  determine  whether  individual  team  member’s 
workload  changed  due  to  the  cockpit  configuration;  (2)  a  separate  survey  question  addressed  their 
perceived  overall  workload;  and  (3)  a  correlation  analysis,  similar  to  the  analysis  completed  for 
information  requirements,  was  performed  using  each  team  member’s  estimations  of  their 
teammate’s  significant  sources  of  workload.  The  findings  for  each  of  these  methods  are 
discussed  below. 
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Table  15. 

ANOVA  for  correlation  matrix,  off  nominal. 
Dependent  Variable:  Correlate 


Source 

df 

MSE 

F 

mm 

Observed 

Power 

Corrected  Model 

1 

.587 

10.803 

.0041 

.874 

Intercept 

1 

4.914 

90.501 

.0000 

1.000 

compart 

1 

.587 

10.803 

.0041 

.874 

Error 

18 

.054 

Total 

20 

Corrected  Total 

19 

Complementary  Information 


Figure  37.  Box  plot  for  Spearman  Rank  Correlation  Coefficient. 

Table  16. 

Binomial  test  for  awareness  of  information  requirements. 


Complementary 

Information 

N 

Observed 

Prop. 

Test  Prop. 

Exact  Sig. 
(2-tailed) 

Clarity  of 

Group  1 

No 

3 

.18 

.50 

.013 

Crewmember's 

Group  2 

Yes 

14 

.82 

Requirements 

Total 

17 

1.00 

Clarity  of  own 

Group  1 

No 

3 

.17 

.50 

.008 

Requirements 

Group  2 

Yes 

15 

.83 

Total 

18 

1.00 
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Individual  workload  ratings  were  assessed  using  the  RTLX  in  Appendix  B  and  analyzed  in 
various  groupings:  aggregated  to  the  team  level,  source,  position,  position  and  source,  by 
configuration  and  by  condition.  The  characteristics  of  the  data  were  consistant  with  the 
assumptions  necessary  to  analyze  these  data  using  ANOVA  techniques.  Therefore  a  GLM 
ANOVA  was  used  for  this  analysis.  Generally,  there  were  minimal  significant  effects  due  to 
information  complementariness  on  team  member’s  workload;  results  of  individual  ratings  by 
position  and  source  are  displayed  in  Table  17.  The  only  significant  change  in  mean  ratings  for 
individual  workload  due  to  the  independent  variable  was  detected  in  the  navigator’s  mental 
workload  in  the  nominal  condition,  which  increased  when  operating  in  the  complementary 
cockpit  configuration.  Additionally,  the  cells  highlighted  in  yellow  (italics  in  light  gray  fields) 
were  marginally  significant;  these  measures  also  increased  in  the  complementary  configuration. 


Table  17. 

Individual  team  member  workload  significance  levels. 


Nominal 

Off-Nominal 

Helicopter 

Helicopter 

Pilot 

Navigator 

Pilot 

|  Mental 

0.700 

0.765 

0.407 

0.694 

0.657 

0.925 

0.053 

|]BJn33ZDSBHH 

0.086 

0.262 

0.500 

0.310 

Effort 

0.613 

0.068 

0.535 

0.054 

Performance 

0.804 

0.874 

0.743 

0.559 

Frustration 

0.165 

0.432 

0.266 

0.380 

Teams  were  asked  how  difficult  it  was  to  coordinate  actions  with  their  teammate. 

Due  to  the  non-parametric  nature  of  these  data,  they  were  analyzed  using  a  Mann-Whitney  test. 
The  results  are  shown  in  Table  18  and  Table  19.  The  p-value  of  .06  was  nearly  significant  (alpha 
=  .05).  The  mean  rank  of  the  complementary  configuration  was  lower  than  the  normal  condition, 
which,  based  on  the  scale  above,  indicated  that  team  members  felt  that  it  was  slightly  more 
difficult  to  coordinate  their  actions  when  operating  in  the  complementary  configuration. 

The  team  member  RTLX  (Appendix  B)  was  used  to  record  team  member  rankings  following 
each  run.  Team  members  were  asked  to  estimate  the  sources  of  workload  for  their  teammate 
using  the  modified  NASA  TLX  scale.  Based  on  their  scores,  the  sources  of  workload  (mental, 
effort,  temporal,  etc.)  were  rank  ordered  (1-6).  Rankings  from  the  pilot  and  navigator  were 
matched,  and  a  correlation  matrix  was  developed  using  the  Spearman  Rank  Order  Correlation 
Coefficient.  Relevant  correlations  were  selected  from  the  matrix  and  were  analyzed  using  a 
GLM  ANOVA  for  significant  differences  in  correlation  between  information  configurations.  For 
instance,  the  pilot’s  ranking  of  the  navigator’s  workload  was  matched  with  the  navigator’s 
ranking  of  the  navigator’s  workload. 

The  ANOVA  performed  with  the  navigator  as  the  estimator  identified  no  significant  results. 
On  the  contrary,  differences  in  the  mean  correlation  coefficients  were  marginally  significant  ip  = 
.069)  when  the  pilot  estimated  the  navigator’s  workload.  The  complete  ANOVA  is  displayed  in 
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Table  20.  Figure  38  illustrates  the  increase  in  the  median  and  mean  from  the  normal 
configuration  to  a  complementary  distribution  of  cockpit  information.  The  inner  quartile  range 
was  decreased  when  the  team  operated  in  the  complementary  configuration  (Figure  39). 


Table  18. 

Mann- Whitney  Test  for  difficulty  level. 


Difficulty 

Mann-Whitney  U 

137.000 

Wilcoxon  W 

347.000 

Z 

-1.880 

Asymp.  Sig. 

.060 

Table  19. 

Mann- Whitney  Ranks  for  difficulty  level. 


Complementary 

N 

Mean  Rank 

Sum  of  Ranks 

Difficulty  No 

20 

23.65 

473.00 

Yes 

20 

17.35 

347.00 

Information  Distribution 

1=Very 

Difficult 

Difficulty  Level 

2=Difficult  3=Easy 

4=Very 

Easy 

Normal 

1 

2 

3 

4 

Complementary 

1 

2 

3 

4 

Figure  38.  Difficulty  level  survey  question. 
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Table  20, 

ANOVA  Pilot  Estimated  Workload  of  Team  Member  Correlation 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Observed 

Power 

Corrected  Model 

.422 

1 

.422 

3.582 

.069 

.447 

Intercept 

5.581 

1 

5.581 

47.328 

.000 

1.000 

complementary 

.422 

1 

.422 

3.582 

.069 

.447 

Error 

3.302 

28 

.118 

Total 

9.306 

30 

Corrected  Total 

3.725 

29 

Participant  effects 

The  first  experiment  was  balanced  within  subjects  to  account  for  order  and  training  effects. 
The  second  experiment  consisted  of  one  experimental  run  similar  to  the  previous  two;  this  was  a 
balanced  between  subjects  design  between  the  two  information  distribution  levels.  Participant 
effects  on  communication  and  performance  were  not  significant  to  the  findings  in  this 
experiment  and  are  detailed  in  Appendix  B. 
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Discussion 


As  stated  in  the  introduction,  the  overarching  objective  of  this  study  was  to  provide 
empirical  evidence  that  providing  task  relevant  information  to  individual  team  members  in  a  time 
critical  environment,  while  limiting  their  access  to  non-relevant  information,  would  improve 
team  performance  and  change  team  interactions  by  developing  complementary  team  mental 
models.  Furthermore,  this  method  of  distributing  information  among  team  members  would 
provide  individual  team  members  with  a  more  accurate  “task  relevant”  mental  model  of  their 
own  environment.  Seven  hypotheses  were  offered  to  test  this  theory  and  are  discussed  below. 

Confirmation  of  experimental  hypotheses 

Hypothesis  1:  Team  and  individual  performance,  during  nominal  conditions,  will  remain 
constant  when  information  is  complementary  when  compared  to  performance  during  nominal 
conditions  in  a  normal  cockpit  configuration. 

The  set  of  performance  measures  for  team  and  individual  performance  was  a  combination  of 
both  objective  and  subjective  metrics.  Individual  performance  measures  showed  no  significant 
changes  in  their  means,  and  total  errors  was  the  only  team  metric  that  showed  a  significant 
difference.  Total  errors  decreased  when  teams  operated  in  a  cockpit  that  was  configured  with 
complementary  information  versus  the  normal  configuration.  Fifteen  errors  were  committed  in 
the  normal  condition  versus  five  in  the  complementary  configuration.  This  indicates  that  team 
members,  under  nominal  conditions,  performed  better  when  the  information  in  their  environment 
was  distributed  in  such  a  manner  that  individual  tasks  and  responsibilities  were  supported  more 
specifically.  In  terms  of  the  hypothesis,  generally  team  and  individual  performance  remained 
constant,  but  total  errors  (process  errors  +  navigation  errors)  decreased. 

Nominal  conditions  are  conditions  that  are: 

•  Highly  proceduralized, 

•  Expected  by  aircrews, 

•  Practiced  routinely, 

•  Operated  in  regularly,  and 

•  Trained  and  cross-trained  for  by  aircrews. 

The  fact  that  performance,  other  than  errors,  remained  constant  is  a  significant  finding.  This 
suggests  that,  in  the  current,  normal  configuration,  there  is  information  presented  to  team 
members  that  is  not  needed,  nor  used,  to  perform  their  individual  tasks  and  team  responsibilities; 
and  when  taken  away,  performance  remained  constant.  In  this  experiment,  team  performance 
actually  improved.  This  finding  supports  the  proposition  that  while  teams  may  need  to  have 
access  to  an  abundance  of  relevant  information,  access  to  additional  information  may  increase 
cognitive  workload,  add  confusion  and  stress,  and  degrade  performance. 
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Hypothesis  2:  Team  performance  during  off-nominal  conditions  will  improve  when  information 
is  complementary  when  compared  to  performance  during  off-nominal  conditions  in  a  normal 
cockpit  configuration. 

To  test  this  hypothesis,  metrics  from  the  nominal  condition  were  augmented  by  six  metrics 
specific  only  to  the  off-nominal  condition.  Most  of  these  additional  metrics  addressed  the  team’s 
handling  of  entry  into  EMC  and  the  conduct  of  dealing  with  an  in-flight  emergency;  both  of  these 
were  unforeseen  events.  In  testing  this  hypothesis,  the  results  revealed  that  individual  and  team 
performance  either  displayed  non-significant  variations  or  improved;  no  measure  of  performance 
worsened  when  information  was  distributed  in  a  complementary  fashion. 

First,  pilot  control  of  the  aircraft  generally  remained  constant  throughout  the  entire  flight 
profile,  except  during  entry  into  IMC.  Task  twelve  of  the  flight  profile  was  the  segment  of  flight 
where  teams  were  exposed  to  IIMC.  During  this  segment,  maintenance  of  airspeed  significantly 
improved  (i.e.,  RMSE  airspeed  decreased)  in  the  complementary  configuration.  The  median 
airspeed  error  decreased  17%  and  the  mean  decreased  14.2%  when  the  distribution  of 
information  was  complementary.  During  entry  into  IMC,  the  pilot’s  primary  task  is  to  maintain 
control  of  the  aircraft.  This  requires  the  pilot  to  focus  on  flight  instrumentation  to  coordinate  a 
smooth  transition  from  Visual  Flight  Rules  (VFR)  to  Instrument  Flight  Rules  (IFR).  During  this 
task,  the  importance  of  airspeed  maintenance  cannot  be  over  stated.  According  to  the  UH-60 
Aircrew  Training  Manual,  immediate  action  steps  are  as  follows: 

1.  Announce  “IMC,”  maintain  proper  aircraft  control,  and  make  the  transition  to  instrument  flight 
immediately. 

2.  Initiate  correct  IIMC  recovery  procedures. 

a.  Attitude  -  Level  the  wings  on  the  attitude  indicator. 

b.  Heading  -  Maintain  heading;  turn  only  to  avoid  known  obstacles  or  as  briefed  for  multi¬ 
ship  operations. 

c.  Torque  -  Adjust  torque  as  necessary. 

d.  Trim  -  Trim  aircraft  as  necessary. 

e.  Airspeed  -  Adjust  airspeed  as  necessary  (in  this  case  airspeed  should  have  been 
maintained  as  a  climb  was  initiated). 

3.  Set  transponder  to  emergency,  as  required. 

4.  Contact  ATC,  as  required.  Comply  with  ATC  instructions,  local  regulations,  and  SOP  (TC  1- 
237:  UH-60  Aircrew  Training  Manual). 

By  providing  the  pilots  with  immediate  access  only  to  information  relevant  to  their  tasks  and 
responsibilities,  they  were  able  to  focus  their  attention  on  the  flight  instruments  and  control  the 
aircraft  better. 

Second,  navigators  were  not  flying  the  aircraft  but  were  responsible  for  a  range  of  other 
tasks;  complete  all  radio  calls,  calculate  ETE,  monitor  systems,  coordinate  navigation  with  pilot, 
etc.  Generally,  performance  of  their  specified  tasks  remained  constant  as  configurations  varied, 
except  for  the  completion  of  required  radio  calls.  Navigators  were  briefed  that  they  were 
responsible  for  a  list  of  required  calls  for  each  leg.  In  the  complementary  configuration. 


74 


navigators  seemed  to  be  able  to  concentrate  on  their  responsibilities  better;  they  did  not  get 
distracted  by  having  to  divert  their  attention  to  non-relevant  information.  This  resulted  in  more 
consistent  and  improved  performance  in  this  individual  task. 

Third,  teams  reacted  better  when  they  were  presented  with  an  in-flight  emergency.  In  the 
complementary  configuration,  all  teams  diagnosed  the  proper  engine  malfunction  and  identified 
the  proper  emergency  procedure;  on  the  contrary,  only  three  out  of  five  were  able  to  achieve  that 
end  in  the  normal  condition.  Additionally,  the  time  required  to  diagnose  the  emergency  was 
significantly  reduced  in  the  complementary  configuration.  The  emergency  presented  to  the  team 
was  a  single  engine  alternator  failure  (#1  engine);  the  #2  engine  was  unaffected  and  the  #1  engine 
was  still  operational.  A  complete  loss  of  engine  alternator  power  for  the  #1  engine  results  in  the 
#1  engine  increasing  to  maximum  power  with  a  loss  of  cockpit  indications  of  engine  RPM  for  #1 
engine,  torque,  Gas  Generator  Speed  (Ng)  Indicator,  and  an  engine  out  audio  and  warning  light. 
Turbine  Gas  Temperature  (TGT)  was  the  only  indication  that  the  #1  Engine  was  still  operating 
normally.  The  warning  in  Figure  40  is  posted  in  the  UH-60  operator’s  manual: 


Do  not  respond  to  engine-out  audio  and 
warning  light  until  checking  TGT,  Ng, 
and  %  RPM  1  and  2. 

Figure  40.  Engine  out  warning. 


Under  normal  conditions  teams  appeared  to  communicate  less  efficiently  during  analysis  of 
the  engine  malfunction.  Both  team  members  commented  on  various  bits  of  information 
presented  in  the  cockpit  in  a  seemingly  unorganized  manner.  Teams  had  been  trained  to 
immediately  check  TGT,  Ng,  and  %  RPM,  yet  this  was  not  done.  In  the  complementary 
configuration,  analysis  of  the  malfunction  seemed  to  be  more  organized.  The  pilots  were  able  to 
maintain  their  focus  on  their  flight  instruments  while  the  navigators  explicitly  shared  the  engine 
information  (TGT,  Ng,  etc.)  with  the  pilots;  the  malfunction  analysis  was  improved  through  a 
system  design  that  forced  explicit  communication.  Both  team  members  were  aware  of  what 
information  each  had  available,  and  more  importantly,  what  information  the  other  team  member 
did  not  have  available  to  them.  This  resulted  in  a  decrease  in  the  time  required  to  diagnose  the 
engine  malfunction  of  nearly  66%  from  the  normal  to  the  complementary  condition.  Teams 
made  faster  decisions  (decrease  in  diagnosis  time)  and  better  decisions  (correct  analysis  of 
emergency).  Future  research  could  include  a  sequential  analysis  of  communications  during  the 
off-nominal  conditions. 

Hypothesis  3:  Explicit,  task  relevant,  verbal  communications  will  increase  when  information  is 
complementary  in  both  nominal  and  off-nominal  conditions  when  compared  to  communications 
in  a  normal  cockpit  configuration. 
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Results  suggested  that  teams  shared  more  task  relevant  information  in  the  complementary 
configuration  than  in  the  normal  configuration.  These  findings  are  important  because  they 
provide  new  insight  into  how  teams  share  information.  Additionally,  these  findings  provide 
empirical  evidence  that  system  design  can  facilitate  an  increase  in  the  explicit  sharing  of 
information  between  team  members.  Communications  were  classified  as  transfers  and  requests 
that  were  task  relevant  or  non-relevant.  Not  only  was  there  a  significant  increase  in  task  relevant 
transfers  of  information,  there  also  was  a  corresponding  decrease  in  non-relevant  transfers.  The 
latter  only  occurred  during  nominal  operations;  non-relevant  transfers  were  minimal  in  both 
configurations  during  off-nominal  operations. 

Furthermore,  during  off-nominal  operations,  which  are  typically  not  as  proceduralized  as 
nominal  operations,  specific  acknowledgements  of  receipt  of  information  increased  significantly 
in  the  complementary  condition.  Team  members  tended  to  acknowledge  each  other’s  commands 
to  ensure  that  they  received  the  proper  information.  For  instance,  when  the  navigator  instructed 
the  pilot  to  “turn  right  to  heading  150...”  the  pilot  would  respond,  “roger,  turn  right  to  150 
degrees.”  This  type  of  specific  acknowledgement  ensures  that  both  team  members  know  “who 
has  what”  information  and  increases  the  likelihood  of  catching  an  error  before  it  occurs.  This 
technique  of  communication  is  encouraged  in  nominal  operations  and  trained  for  in  other  than 
nominal  operations.  For  example,  the  following  is  an  excerpt  from  the  UH-60  aircrew  training 
manual  concerning  team  coordination  procedures: 

Procedures:  The  call  and  response  method  should  be  used.  The  team  member  reading  the 
checklist  will  read  the  complete  checklist  item.  The  team  member  performing  the  check  will 
answer  the  appropriate  response.  For  example,  for  the  call  “Anti-collision/position  lights  - 
As  required”  the  response  might  be  “Anti-collision  lights,  both,  night;  position  lights,  steady, 
bright.”  Responses  that  don’t  clearly  communicate  action  of  information  should  not  be  used. 
For  example,  when  responding  to  the  call,  “Systems  -  Check”  replying  with,  “Check” 
doesn’t  clearly  indicate  that  the  systems  are  within  the  normal  operating  range.  A  response 
of  “All  in  the  normal  operating  range”  communicates  more  accurate  information  (Training 
Circular  1-237,  Aircrew  Training  Manual,  August  2004). 

This  is  also  the  required  method  of  communication  when  team  members  transfer  the  flight 
controls  from  one  pilot  to  the  other;  this  is  referred  to  as  a  three  way  positive  transfer  of  controls: 

Pilot:  You  have  the  controls. 

Navigator:  I  have  the  controls. 

Pilot:  You  have  the  controls. 

By  designing  the  distribution  of  information  in  a  complementary  manner,  specific 
acknowledgements  of  information  were  increased  during  critical  operations  in  off-nominal 
conditions.  This  type  of  system  design  complements  current  training  and  procedures  practiced  in 
this  and  other  domains. 

These  results  indicated  that  when  teams  shared  more  task  relevant  information  they 
performed  significantly  better  (i.e.,  made  fewer  errors,  decreased  decision  time,  had  better 
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aircraft  control,  and  had  more  effective  radio  communications)  than  when  they  shared  less  task 
relevant  information.  Thus,  these  results  provide  some  empirical  support  that  complementary 
team  mental  models  may  be  a  means  of  improving  coordinated  performance  during  nominal  and 
off-nominal  conditions. 

Hypothesis  4:  The  team ’s  task  relevant  anticipation  ratio  will  increase  when  information 
distribution  is  complementary  when  compared  to  operations  in  a  normal  cockpit  configuration. 

Based  on  the  results  of  Experiment  1,  the  task  relevant  anticipation  ratio  was  expected  to 
increase  when  information  was  complementary.  However,  results  indicated  the  opposite;  there 
was  a  significant  decrease  in  this  measure  during  nominal  and  off-nominal  operations.  Teams  in 
this  experiment  provided  higher  rates  of  information,  but  the  decreased  anticipation  ratio 
indicates  that  team  members  did  not  anticipate  the  information;  information  was  requested  at  an 
even  higher  rate.  The  anticipation  ratio  is  calculated  as: 

( Transfers  of  Task  Relevant  Information  +  Transfers  of  Action  +  Total 
Acknowledgements)  /Requests  for  Task  Relevant  Information 

Investigation  to  the  possible  reasons  for  this  unexpected  behavior  uncovered  some  interesting 
explanations. 

Although  literature  has  provided  empirical  data  suggesting  that  teams  that  provide  higher 
rates  of  information  in  advance  perform  better  under  various  conditions  (Entin  and  Entin,  2001 ; 
Stout  et.  al.,  1999),  Orasanu  (1994)  observed  quite  the  opposite  when  dealing  with  some 
effective  teams.  Among  two  member  teams  analyzed  by  Orasanu  and  Fischer  (1992),  captains  of 
higher  performing  teams  talked  less  during  the  high  workload  phase  of  flight  than  during  normal 
operations  (Orasanu,  1994),  which  could  account  for  the  decrease  in  the  off-nominal  situation 
during  the  analysis  of  the  engine  malfunction. 

Furthermore,  care  was  taken  to  configure  the  cockpits  in  a  manner  that  provided  the 
information  required  for  individual  tasks  to  the  responsible  team  member.  This  would  enable 
team  members  to  perform  their  individual  tasks  (i.e.,  control  of  the  aircraft)  without  having  to 
request  information  from  another  team  member.  This  was  attempted  through  a  thorough  task  and 
domain  analysis.  However,  review  of  the  videos  of  team  communications  revealed  that  one  of 
the  information  sources  used  frequently  by  the  pilot  to  fly  (engine  torque  percentage  display)  was 
not  available  to  the  pilot  in  the  complementary  configuration.  This  display  was  grouped  with 
other  engine  instruments  that  the  navigator  was  required  to  monitor.  This  resulted  in  the  pilot 
continuously  asking  for  information  from  the  navigator  concerning  engine  torque,  which 
increased  the  requests  for  information  and,  therefore,  decreased  the  anticipation  ratio.  This 
finding  helped  to  identify  a  flaw  in  the  design  process.  Even  though  a  thorough  task  and  domain 
analysis  was  completed,  to  include  interviews  with  subject  matter  experts  and  personal 
experience  of  the  analyst,  real  time  observation  of  the  environment  and  the  task  is  important 
during  development.  People  are  innovative  and  will  find  uses  for  artifacts  for  which  they  were 
not  originally  designed;  observation  must  be  included  as  an  integral  step  in  designing  or 
redesigning  a  system. 
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A  third  reason  that  the  ratio  decreased  was  due  to  the  lack  of  structure  of  the  environment 
and  task.  Pilots  frequently  request  their  navigator  to  repeat  or  update  information  concerning 
altitude,  airspeed  and  heading.  This  is  in  contrast  to  the  initial  experiment  that  was  conducted  in 
an  automobile  simulator.  In  the  automobile  experiment,  participants  were  required  to  drive 
speed  limits  as  posted  on  the  road  so  there  was  no  need  to  ask  their  navigator  for  speed 
information.  Furthermore,  once  the  navigator  gave  an  instruction  to  turn  on  a  specific  road,  the 
driver  easily  maintained  that  “heading”  by  staying  on  the  road;  finally,  there  was  no  altitude 
concern  in  the  automobile.  In  the  helicopter,  the  pilots  tended  to  ask  for  updates  to  these 
parameters  to  ensure  that  they  were  maintaining  the  proper  route  guidance.  This  also  increased 
the  denominator  in  the  calculation  of  the  anticipation  ratio. 

Finally,  it  was  significant  that  all  measures  of  the  ratio’s  variation  were  decreased  in  the 
complementary  configuration.  Even  though  the  ratio  was  lower,  decreased  variation  indicated  a 
more  consistent  communication  pattern  in  both  nominal  and  off-nominal  operations.  This 
suggests  that  the  configuration  of  the  cockpit  had  the  same  effect  on  all  teams  operating  in  that 
environment.  Whereas  the  large  variation  in  the  ratio  in  the  normal  configuration  indicates  that 
some  teams  had  low  ratios  and  some  high;  the  teams  were  not  consistent  throughout.  This  is 
significant  when  applied  to  many  domains  in  which  teams  and  individuals  rotate.  In  these 
environments  the  same  team  members  are  not  always  present  or  performing  the  same  tasks; 
therefore,  performance  is  likely  to  improve  if  the  system  configuration  supports  consistency  in 
communications  among  teams. 

Hypothesis  5:  Team  member  information  requirement  rankings  will  be  more  similar  when 
information  is  complementary  when  compared  to  rankings  elicited  while  operating  in  a  normal 
cockpit  configuration. 

This  hypothesis  tested  whether  teams  demonstrated  the  ability  to  determine  the  information 
requirements  of  their  teammate  more  accurately  in  the  complementary  configuration  than  in  the 
normal  configuration.  Using  standard  phases  of  flight  (takeoff,  landing,  etc.)  to  delineate 
segments  of  each  mission  for  assessment,  team  members  were  able  to  predict  the  information 
required  by  their  teammate  better  while  operating  in  the  complementary  configuration.  This  is  a 
significant  finding  and  suggests  the  complementary  configuration  develops  a  more  accurate 
mental  model  of  team  information  requirements;  they  were  able  to  form  accurate  expectations 
and  explanations  of  informational  requirements.  This  study  defines  this  state  as  complementary 
team  mental  models.  Therefore,  the  development  of  this  type  of  team  mental  models  was 
supported  when  the  information  was  displayed  in  a  complementary  manner  among  team 
members.  Team  members  were  able  to  use  these  expectations  and  explanations  to  provide  useful 
information  to  the  other  team  member  during  critical  phases  of  their  mission  (i.e.,  during  the 
diagnosis  of  an  engine  malfunction). 

When  asked  explicitly  which  configuration  supported  better  coordination  of  team  activities 
and  identification  of  individual  and  team  member  informational  requirements,  participants 
suggested  that  the  complementary  condition  was  more  suitable.  This  result  suggests  that  not  only 
were  the  team  member  estimates  of  requirements  more  accurate,  but  so  were  the  individuals’ 
assessments  of  their  own  information  requirements.  With  the  abundance  of  information 
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available  in  a  helicopter  cockpit,  this  design  method  is  simply  a  filtering  device  that  aids  in  both 
individual  task  performance  and  team  coordination. 

Hypothesis  6:  Individual  workload  ratings  will  remain  constant  between  information  levels 
regardless  of  cockpit  configuration. 

Generally,  individual  workload  remained  constant  across  configurations  except  for  the 
navigator’s  mental  workload.  The  navigator’s  mental  workload  increased  when  exposed  to  the 
complementary  cockpit  configuration.  Additionally,  the  navigator’s  effort  was  borderline 
significant  relative  to  the  5%  level  (p  =  .068);  this  measure  also  tended  to  increase  in  the 
complementary  configuration.  It  is  understandable  that  team  members  might  experience  an 
increase  in  workload  because  they  are  operating  in  an  unfamiliar  environment  for  the  first  time. 
The  limited  increases  in  workload  suggest  that  the  workload  would  most  likely  decrease  in 
subsequent  exposures  to  this  type  of  configuration.  This  effect  should  be  studied  more  in-depth 
in  future  investigations. 

Hypothesis  7:  Team  member  source  of  workload  rankings  will  be  more  similar  when  information 
is  complementary  when  compared  to  rankings  during  operations  in  a  normal  cockpit 
configuration. 

Teams  were  not  found  to  estimate  the  ranks  of  their  teammates’  sources  of  workload  better 
in  the  complementary  condition.  The  analysis  showed  that  the  pilot’s  ability  to  estimate  the 
navigator’s  workload  increased  slightly  but  the  results  were  not  significant  (p  =  .069).  Both  the 
inner  quartile  range  and  variance  showed  a  slight  decrease  that  indicates  that  pilots  might  be 
more  consistent  in  their  accuracy.  The  analyst,  using  the  participants’  score  for  each  source  of 
workload,  determined  the  rankings  that  were  used  for  this  analysis.  This  was  an  indirect  measure 
of  ranking  workload  sources.  In  the  future,  research  participants  could  be  asked  to  explicitly  rank 
order  the  sources  of  workload  to  indicate  which  source  is  causing  the  most  workload  to  the  least. 

Summary 

The  findings  of  this  research  provide  new  insights  into  how  the  distribution  of  information 
among  team  members  effects  the  development  of  team  mental  models,  team  and  individual 
performance,  and  communications  that  have  not  been  empirically  documented  elsewhere.  The 
results  are  applicable  to  a  variety  of  domains  where  teams  are  operating  in  a  complex 
environment.  The  results  of  this  experiment  provide  empirical  evidence  that  providing  task 
relevant  information  to  individual  team  members  in  a  time  critical  environment,  while  limiting 
their  access  to  non-relevant  information,  improves  individual  and  team  performance  by  changing 
team  interactions  and  helping  to  develop  complementary  team  mental  models.  Furthermore,  this 
method  of  distributing  information  among  team  members  provides  individual  team  members 
with  a  more  accurate  “task  relevant”  mental  model  of  their  own  environment. 
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Conclusions 


This  study  demonstrates  that  a  complementary  information  distribution  among  team 
members  can  promote  improved  team  process  and  performance.  Although  not  directly 
measurable,  this  improvement  is  likely  a  result  of  teams  developing  more  efficient  team  mental 
models,  i.e.,  complementary  team  mental  models.  Complementary  team  mental  models  provide 
a  condition  in  which: 

1.  Each  team  member  has  the  knowledge  necessary  to  conduct  his/her  tasks,  as  may  be 
measured  by: 

•  Individual  performance  and 

•  Direct  assessment  of  clarity  of  individual  information  requirements. 

2.  Each  team  member  knows  which  information  is  available  to  the  other  team  members  should 
he/she  need  to  seek  it,  as  may  be  measured  by: 

•  Team  performance, 

•  Direct  assessment  of  the  clarity  of  team  member  information  requirements, 

•  Estimation  of  team  member’s  sources  of  workload,  and 

•  Task  relevant  transfers  of  information. 

3.  Each  team  member  knows  which  information  is  needed  from  them  to  other  team  members 
and  when,  as  may  be  measured  by: 

•  Team  performance, 

•  Communication  rates,  and 

•  Task  anticipation  ratio. 

The  distribution  of  complementary  information  underpinned  this  thesis.  Hence,  the  precise 
determination  of  information  sources  available  in  each  domain  and  the  information  required  by 
team  members  to  accomplish  their  specified  task  is  critical.  This  thesis  proposes  that  alignment 
of  information  sources  with  task  requirements  can  be  accomplished  though  a  three  phase  method: 
(1)  work  domain  analysis  of  the  information  available  in  the  work  environment,  (2)  analysis  of 
the  information  requirements  of  specific  tasks,  and  (3)  a  mapping  of  information  sources  to  their 
corresponding  tasks.  The  work  domain  analysis  identifies  the  structure  of  the  environment  where 
work  takes  place;  this  phase  produces  an  Abstraction-Decomposition  Space  (ADS)  (Vicente, 
1999),  also  known  as  the  Abstraction  Hierarchy  (AH)  (Rasmussen,  1985).  Analysis  of  the  tasks, 
the  second  phase,  is  most  appropriately  accomplished  by  conducting  separate  Hierarchical  Task 
Analyses  (HTA),  thereby  providing  an  efficient  method  of  describing  the  task  by  developing  only 
the  parts  of  the  hierarchy  that  are  needed  for  the  scope  of  the  design  (Kirwan  and  Ainsworth, 
1992).  The  last  phase  maps  the  information  sources  available  in  each  domain  to  the  individual 
and  team  tasks  that  they  support. 

The  method  described  above  was  then  applied  to  two  separate  domains:  driving  and  flying. 
The  results  identified  the  physical  forms  of  information  sources  required  to  conduct  the  task  of 
navigation  in  both  domains;  these  sources  are  tabulated  in  Figure  41.  The  availability  of  these 
information  sources  to  each  of  the  team  members  determined  the  level  of  complementariness  of 
information,  the  main  independent  variable  for  experimental  purposes.  The  method  of 
determining  the  information  requirements  is  a  unique  application  of  two  separate  analytical 
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processes,  work  domain  analysis  and  hierarchical  task  analysis,  and  is  recommended  when  the 
analyst  requires  a  descriptive  analysis  of  a  specified  task  conducted  by  a  small  team  in  a  complex 
domain. 


Automobile 

Helicopter 

Driver/Pilot 

Driving  instruments 

Flight  Instruments 

Window  View 

Windscreen  View 

System  Health/Status  Indicators 

System  Health/Status  Indicators 

Navigator 

Clock 

Clock 

Maps 

Maps 

Global  Positioning  System 

Crew 

Internal  Lights 

Voice  communication  Radios 

Internal  Lights 

Figure  41.  Information  sources  required  for  both  domains. 


Over  the  course  of  two  experiments,  the  complementariness  of  team  mental  models  and  its 
benefits  was  measured  through  a  variety  of  dependent  variables.  The  data  were  categorized  into 
four  groups:  performance,  communications,  workload,  and  information  requirements.  Although 
complementary  team  mental  models  cannot  be  measured  directly,  their  existence  can  be  inferred 
through  a  combination  of  these  metrics  (Cooke  et  al.,  in  press;  Entin  and  Entin,  2001;  Langan- 
Fox,  Code,  and  Langfield-Smith,  2000;  Rouse,  Cannon-Bowers,  and  Salas,  1992).  The  process 
and  outcome  measures  corroborate  that  complementary  team  mental  models  may  help  clarify 
decision-making  roles  and  responsibilities,  individual  and  team  member  information 
requirements,  increase  the  efficiency  of  explicit  communications,  and  improve  performance. 

The  conditions  that  teams  operated  in  during  this  research  were  separated  in  two  categories: 
nominal  conditions  and  off-nominal.  During  nominal  conditions,  there  were  no  inconsistencies 
between  instructions  given  and  participant’s  ability  to  follow  the  route  in  the  simulator.  During 
off-nominal  conditions  in  the  automobile  simulator,  teams  encountered  a  roadblock  that  had  to 
be  bypassed;  in  the  helicopter  domain,  teams  experienced  EMC  and  a  single  engine  alternator 
failure  during  the  flight.  The  results  of  these  experiments  are  summarized  below. 

Performance 


Individual  and  team  performance  was  assessed  through  objective  and  subjective  metrics 
during  nominal  and  off-nominal  conditions.  Figure  42  displays  dependent  variables  that  showed 
significant  effects  due  to  changes  in  the  distribution  of  information  sources.  The  original 
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hypothesis  proposed  that  during  nominal  conditions,  individual  and  team  performance  would 
remain  constant;  performance  was  hypothesized  to  only  improve  in  the  off-nominal  condition. 
Results  indicated  that  team  performance  also  improved  in  the  nominal  condition  in  the  presence 
of  complementary  information.  Increases  in  team  performance  were  shown  through  decreased 
decision  making  time  and  improved  the  quality  of  decisions.  Furthermore,  individual 
performance  of  helicopter  pilots  also  improved  during  off-nominal  conditions. 
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Figure  42.  Summary  of  performance  effects  due  to  a  complementary 
distribution  of  information  sources. 


Communications 


In  both  domains,  a  complementary  information  distribution  resulted  in  an  increase  in  the  rate 
of  transfers  of  task  relevant  information,  acknowledgements  and  total  communications. 
Additionally,  the  rate  of  transfers  of  non-relevant  information  decreased  in  all  conditions. 
Although  not  the  focus  of  this  study,  it  should  be  noted  that  communication  rates  decreased 
overall  in  the  off-nominal  condition  when  compared  to  the  nominal  condition  in  both  distribution 
configurations. 

Requests  were  recorded  when  either  team  member  explicitly  asked  for  information  or  action 
from  the  other  team  member.  Generally  requests  were  for  information  regarding  the  team  task 
(i.e.,  current  or  future  state  of  vehicle);  there  were  relatively  few  requests  for  non-relevant 
information.  There  was  no  significant  difference  in  team  requests  between  a  normal  and 
complementary  configuration  within  the  automobile  domain,  but  a  significant  increase  in  team 
requests  for  action  did  occur  in  the  helicopter  domain. 

In  both  nominal  and  off-nominal  conditions  in  both  domains,  transfers  of  task  relevant 
information  increased  in  the  complementary  configuration  when  compared  to  the  normal 
configuration  (Table  21).  It  also  is  important  to  note  that  transfers  of  non-relevant  information 
decreased  in  the  nominal  condition  in  both  domains;  in  the  off-nominal  condition  in  the 
helicopter  domain,  there  were  minimal  transfers  in  either  condition.  Furthermore,  total  team 
transfers  increased  in  both  domains  under  all  conditions  in  the  complementary  configuration. 
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Table  21. 

Summary  of  communication  effects  due  to  complementary 
distribution  of  information  sources. 


Significant  Diference  Between  Configurations 


Automobile 

Helicopter 

Nominal 

Nominal 

Off-Nominal 

Task  Info 

NO 

NO 

NO 

Non-Task 

Info 

NO 

N/A 

NO 

Total  Info 

NO 

NO 

NO 

Action 

NO 

YES. 

1 

Task  Info 

MARGINALLY 

Non-Task 

Info 

mim 

NO 

Action 

NO 

NO 

Total ' 

Wmmi 

1 

General 

NO 

NO 

NO 

Specific 

MARGINALLY 

YES 

f: 

Total 

YES 

NO 

NO 

;■  ; 

YES 

Anticipation 

Ratio 

•  •  -S'... 

YES 

;  1 

There  was  a  significant  increase  in  specific  acknowledgements  as  information  became  more 
complementary  in  both  domains,  across  conditions  (Note:  the  effect  of  complementary 
information  in  the  nominal  condition  in  the  helicopter  domain  was  marginally  significant.)  Total 
communication  was  a  combination  of  requests,  transfers,  and  acknowledgements.  Total 
communications  significantly  increased  as  information  became  more  complementary.  This 
measure  indicated  that  the  team  verbalized  more  information.  When  combined  with  the 
significant  increase  in  task  relevant  information  and  decrease  in  non-relevant  information,  this 
increase  was  largely  based  on  task  relevant  transfers. 
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The  task  relevant  anticipation  ratio,  which  describes  the  ratio  of  total  task  relevant 
communications  to  information  requests,  resulted  in  mixed  effects  across  domains.  In  the 
automobile  domain,  the  ratio  increased  significantly  in  the  complementary  configuration,  while 
in  the  helicopter  domain,  the  ratio  decreased.  There  are  three  probable  causes  for  this  dramatic 
difference  in  results.  (1)  Among  some  high  performing  two  member  teams  analyzed  by  Orasanu 
and  Fischer  (1992),  captains  of  higher  performing  teams  talked  less  during  the  high  workload 
phase  of  flight  than  during  normal  operations  (Orasanu,  1994).  This  phenomenon  did  occur  with 
pilots  during  off-nominal  conditions  when  compared  to  nominal  conditions.  (2)  In  the 
experiment  in  the  helicopter,  one  of  the  pilot’s  gauges  used  for  flight  maintenance  was 
erroneously  grouped  with  the  navigator’s  engine  instruments  causing  some  of  the  pilots  to 
continuously  request  information  from  that  instrument  during  normal  operations.  (3)  Differences 
in  the  structure  of  the  domains  (e.g.,  road  structure  vs.  air  route)  may  have  encouraged  pilots  to 
ask  for  repeated  instructions  more  than  the  automobile  teams.  For  example,  shortly  after 
changing  heading,  airspeed,  and  altitude,  aircrews  tended  to  ask  the  navigator  to  verify  these 
parameters  to  ensure  that  they  were  maintaining  the  proper  route  guidance;  this  is  not  necessary 
on  a  road  structure. 

Workload 


Individual  workload  ratings  were  assessed  using  the  RTLX.  The  p-values  for  the  significant 
and  marginally  significant  sources  of  workload  due  to  changes  in  information  distribution  are 
highlighted  in  Table  22.  Generally,  there  were  few  significant  changes  in  team  members’ 
workload. 


Table  22. 

Individual  team  member  workload  p-values  due  to  complementary  information  distribution. 


Nominal 

Off-Nominal 

?..  Automobile 

Helicopter 

Automobile 

Helicopter 

Driver 

Navigator 

Pilot 

Navigator 

Driver 

Navigator 

Pilot 

Navigator 

Mental 

0.871 

0.326 

0.700 

k  0.015 

0.748 

0.664 

0.765 

0.407 

Physical 

0.544 

0.490 

0.694 

0.657 

0.568 

0.742 

0.925 

0.053 

Temporal 

0.828 

0.437 

0.086 

0.262 

0.592 

0.914 

0.500 

0.310 

Effort 

0.881 

mwz 

0.613 

0.068 

0.334 

0.830 

0.535 

0.054 

Performance 

0.699 

0.271 

0.804 

0.874 

0.914 

1.000 

0.743 

0.559 

Frustration 

0.903 

0.178 

0,165 

0.432 

0.668 

0.238 

0.266 

0.380 

•  Red  (dark  gray)  cells  =  significant  effect 

•  Yellow  (light  gray)  cells  and  Italicized  Text  =  marginally  significant  effect 

At  the  end  of  each  experimental  run,  participants  were  given  an  opportunity  to  rate  their 
workload,  specifically  the  difficulty  in  coordinating  with  their  teammate.  Within  the  automobile 
domain,  participants  rated  the  complementary  distribution  significantly  more  difficult  to 
coordinate  actions;  in  the  helicopter  domain,  this  measure  was  marginally  significant.  Overall, 
participants’  answers  to  these  questions  indicated  that  team  members  felt  that  it  was  slightly 
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more  difficult  to  coordinate  their  actions  when  operating  in  the  complementary  configuration; 
however,  this  was  not  corroborated  by  the  TLX  workload  ratings. 


A  team  member  workload  ranking  correlation  analysis  was  conducted  in  the  helicopter 
experiment.  At  the  end  of  each  run,  team  members  estimated  the  workload  of  their  teammate 
using  the  Raw  Task  Load  Index.  Based  on  their  scores,  the  sources  of  workload  were  rank 
ordered  and  a  correlation  matrix  of  the  rankings  from  the  pilot  and  co-pilot  was  developed.  With 
the  navigator  as  the  predictor,  differences  due  to  information  distribution  were  not  significant.  In 
contrast,  when  the  pilot  predicted  the  navigator’s  source  of  workload,  differences  in  the  mean 
correlation  coefficients  were  marginally  significant.  The  median  and  mean  correlation  increased 
with  the  complementary  distribution  of  cockpit  information. 

Information  requirements 

Two  methods  were  used  in  the  helicopter  experiment  to  explicitly  determine  whether  team 
members  felt  they  (1)  had  the  knowledge  necessary  to  conduct  his/her  tasks,  (2)  knew  which 
information  was  available  to  the  other  team  member  should  he/she  need  to  seek  it,  and  (3)  knew 
which  information  was  needed  from  them  to  the  other  team  member,  and  when.  The  first  method 
was  a  similarity  rating  of  team  member’s  information  requirements  during  different  phases  of 
flight;  the  second  was  a  survey  question  at  the  end  of  the  experiment  that  specifically  addressed 
clarity  of  information  requirements  during  complementary  and  normal  cockpit  configurations. 

At  the  end  of  each  run,  team  members  ranked  the  importance  of  each  type  of  information 
available  during  different  phases  of  flight.  In  the  nominal  condition,  no  significant  differences 
were  found  between  the  two  types  of  information  distribution.  In  contrast,  in  the  off-nominal 
condition,  the  correlation  between  pilot  and  navigator  rankings  of  information  improved 
significantly  from  the  normal  configuration  to  a  complementary  distribution  of  cockpit 
information. 

The  following  questions  were  asked  of  each  participant  at  the  end  of  the  helicopter 
experiment: 

1.  In  which  condition  were  you  more  aware  of  your  team  member’s  information  requirements 

(normal  or  complementary)? 

2.  In  which  condition  were  your  own  information  needs  more  clear  (normal  or  complementary)? 

Over  eighty  percent  of  the  participants  that  answered  the  survey  question  indicated  that  the 
complementary  distribution  of  information  within  the  cockpit  provided  more  clarity  to  their  own 
information  requirements,  and  they  felt  they  understood  their  team  member’s  information 
requirements  better.  A  binomial  test  was  used  to  analyze  the  responses;  the  test  proportion  used 
for  the  analysis  was  .5  (50%);  both  response  rates  were  significantly  different  from  chance. 
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Discussion 


Team  members  always  share  information  within  team  environments.  Historically,  the  term 
“share”  has  meant  in  this  context  “to  be  common,”  i.e.,  all  information  sources  in  the 
environment  available  to  all  team  members.  This  thesis  views  the  term  “to  share”  as  "to  divide;” 
for  example,  to  share  the  workload  (Cooke  et  ah,  2000).  In  this  case,  each  team  member  has 
access  to  and  is  responsible  for  the  exchange  of  a  portion  of  the  information  sources  available  to 
him  or  her,  in  addition  to  having  a  portion  common  to  the  team.  It  is  through  this  manner  of 
“sharing”  information  that  this  study  hypothesizes  that  team  mental  models  become  more 
accurate  and  efficient.  They  will  be  more  accurate  because  the  information  sources  are  designed 
with  the  task  and  team  member  roles  as  a  primary  concern:  They  will  be  more  efficient  because 
team  member’s  information  sources  are  distributed  so  that  they  will  mutually  supply  each  other’s 
lack.  This  research  proposed  that  a  complementary  information  distribution  leads  to 
“complementary  team  mental  models,”  a  condition  where  (1)  each  team  member  has  the 
knowledge  necessary  to  conduct  his/her  tasks,  (2)  each  team  member  knows  which  information  is 
available  to  the  other  team  member  should  he/she  need  to  seek  it,  and  (3)  each  team  member 
knows  which  information  is  needed  from  them  to  other  team  members  and  when.  The  empirical 
evidence  corroborates  the  following,  as  discussed  in  the  next  section: 

1.  Development  of  Complementary  Mental  Models:  Team  centered,  complementary 
configurations  of  information  sources  in  correspondence-driven,  complex  domains  will 
improve  the  accuracy  and  efficiency  of  shared  mental  models  by  developing  complementary 
team  mental  models. 

2.  Complementary  Team  Mental  Models  Effect  on  Performance  and  Communications: 
Complementary  team  mental  models  enhance  performance  and  communications  in  both 
nominal  and  off-nominal  operations. 

3.  Complementary  Team  Mental  Models  Effect  on  Team  Workload:  Complementary 
configurations  will  not  adversely  affect  team  workload. 

Development  of  complementary  mental  models 

Corroborating  the  development  of  complementary  mental  models  as  defined  in  this  report 
requires  the  following  conditions: 

Team  members  have  the  knowledge  necessary  to  conduct  their  tasks  better  in  the  complementary 
information  configuration  as  compared  to  a  normal  information  configuration. 

The  experiments  provide  clear  supporting  evidence  of  increased  individual  performance  and 
understanding  of  individual  information  requirements  with  a  complementary  information 
distribution.  As  Figure  53  summarizes,  performance  either  stayed  consistent  or  improved  in  each 
domain.  Improved  performance  was  most  evident  when  the  workload  was  higher,  i.e.,  during 
off-nominal  conditions. 

Team  members  were  more  aware  of  their  team  member’s  information  requirements  in  the 
complementary  information  configuration  as  compared  to  the  normal  configuration. 
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Team  performance  improved  during  both  nominal  and  off-nominal  conditions  due  to  the 
complementary  configuration  of  their  environment.  Likewise,  with  the  complementary 
configuration,  individuals  were  more  aware  of  their  team  member’s  information  requirements, 
and  individuals  explicitly  indicated  that  they  understood  their  team  member’s  information 
requirements  better.  This  increased  understanding  is  invaluable  to  implicit  and  explicit 
coordination  and  can  be  seen  by  the  fact  that  communication  transfers  of  task  relevant 
information  increased  when  teams  operated  in  the  complementary  configuration. 

Team  members  could  anticipate  when  information  was  needed  from  them  to  their  team  member 
better  in  the  complementary  information  configuration  when  compared  to  a  normal  information 
configuration. 

As  noted  in  the  background,  the  most  effective  teams  seem  to  share  their  mental  picture  of 
the  situation  with  other  team  members  (Stout  et  al.,  1999).  These  shared  mental  models  help 
team  members  anticipate  the  needs  of  others,  which  correspondingly  permits  them  to  either 
provide  assistance,  as  it  is  required,  or  to  predict  and  pre-empt  the  need  for  assistance  (Martin 
and  Flin,  1997).  Shared  mental  models  provide  team  members  with  a  common  understanding  of 
who  is  responsible  for  what  task,  what  the  information  requirements  are,  and  allow  team 
members  to  anticipate  one  another’s  needs  so  that  they  can  work  in  sync  and  adjust  their  behavior 
accordingly  (Smith-Jentsch,  Johnston,  and  Payne,  1998). 

Hence,  the  presence  of  a  high  anticipation  ratio,  in  concert  with  other  indicators,  can  be  an 
indicator  of  the  efficiency  of  teams’  shared  mental  models.  Analysis  of  the  two  domains 
examined  here  produced  mixed  results.  In  the  automobile  domain,  there  was  a  significant 
increase  in  this  ratio  indicating  that  anticipation  of  team  member  needs  increased  in  a 
complementary  configuration,  while  the  aircrews  produced  a  significant  decrease.  There  are 
indications  that  an  oversight  in  the  experimental  procedure  may  have  affected  the  aircrew’s  ratio. 

Complementary  team  mental  models  and  performance 

Mathieu  et  al.  (2000)  claim  the  definitive  function  of  shared  mental  models  is  to  allow  team 
members  to  draw  on  their  own  knowledge  as  a  basis  to  decide  on  actions  that  are  consistent  and 
synchronized  with  those  of  their  teammates.  This  research  suggests  that  the  quality  of 
complementariness  in  team  mental  models  enables  team  members  to  do  this  more  effectively. 

The  benefits  of  team  mental  models  are  unmistakably  evident  when  a  team  is  conducting  a 
complex  task,  especially  in  conditions  of  high  workload  (Cannon-Bowers,  Salas,  and  Converse, 
1993;  Rouse,  Cannon-Bowers,  and  Salas,  1992).  Recent  studies  show  that  team  performance  at 
tasks  requiring  anticipation  of  team  member’s  actions  and  information  requirements  is  improved 
with  shared  mental  models  (Espinosa  et  al.,  2001). 

Performance  was  assessed  in  both  domains  in  both  nominal  and  off-nominal  conditions. 
Nominal  conditions  are  generally  highly  proceduralized,  expected  by  team  members,  practiced 
routinely,  operated  in  regularly,  and  trained  and  cross-trained  for  by  teams.  In  nominal 
situations,  individual  performance  generally  remained  constant,  but  procedural  and  navigational 
errors  in  both  domains  decreased  in  the  complementary  configuration.  This  suggests  that  in 
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normal  configurations  there  is  information  presented  to  team  members  that  is  neither  needed  nor 
used  to  perform  their  individual  tasks  and  team  responsibilities.  When  this  superfluous 
information  is  taken  away,  performance  remained  constant  or  improved.  While  teams  may  need 
to  have  access  to  an  abundance  of  relevant  information,  access  to  additional  information  may 
increase  workload,  add  confusion,  and  degrade  performance. 

Off-nominal  conditions  occur  when  an  unanticipated  event  takes  place.  The  value  of  shared 
mental  models  is  greater  during  off-nominal  conditions  where  the  importance  of  adaptation  to 
change  and  anticipation  of  team  member  needs  increases.  During  off-nominal  operations,  teams 
displayed  a  higher  level  of  team  and  individual  performance.  The  effectiveness  of  teams  in  all 
aspects  of  performance  improved  in  the  complementary  configuration  when  compared  to  the 
normal  configuration.  Teams  were  able  to  concentrate  on  the  information  relevant  to  their  role  in 
the  decision  making  process.  For  example,  in  the  automobile  domain,  as  teams  approached  a 
roadblock,  the  navigator  examined  the  map  for  alternate  routes  while  the  driver  concentrated  on 
maneuvering  and  clearing  the  vehicle.  Likewise,  when  aircrews  encountered  IIMC,  the  pilot 
concentrated  on  maintaining  aircraft  control  while  the  navigator  verified  the  minimum  safe 
altitude  and  made  the  appropriate  radio  calls. 

Complementary  team  mental  models  and  communications 

Communications  and  team  operations  are  inseparable.  The  very  definition  of  teams  dictates 
that  individuals  exchange  information  (Cannon-Bowers  and  Salas,  1997).  Communication  is 
more  than  an  exchange  of  information:  it  is  a  means  by  which  teams  coordinate  resources  and 
activities  (Entin  and  Serfaty,  1999),  construct  and  maintain  shared  mental  models  (Orasanu, 
1990),  and  establish  and  maintain  situational  awareness  (Prince  and  Salas,  1993).  The  literature 
indicates  a  positive  relationship  between  communication  frequency  and  increased  performance, 
specifically  in  aircrews  (e.g.,  Roberts  and  O’Riley,  1976;  Foushee  and  Manos,  1981);  the 
research  described  here  provides  additional  empirical  evidence  that  supports  this  theory.  In  the 
experiments  conducted  for  this  study,  the  steady  increase  in  transfers  of  task  relevant  information 
across  conditions  (nominal  and  off-nominal)  in  the  complementary  configuration  reflected  a 
greater  amount  of  task  relevant  information  sharing  between  team  members  in  both  domains. 

This  information  supported  individual  tasks  (e.g.,  maintenance  of  flight  route)  and  team  tasks 
(e.g.,  decision  making).  Furthermore,  the  exchange  of  information  ensures  that  (1)  the  team 
member  that  needs  the  information  has  the  information  and  (2)  both  team  members  know  that  the 
relevant  information  has  been  received.  The  latter  is  further  supported  by  the  consistent  increase 
in  acknowledgements  of  receipt  of  information  across  conditions  in  both  domains  examined  here. 

Complementary  team  mental  models  and  team  workload 

Workload  ratings  were  used  for  two  purposes:  (1)  to  indicate  changes  in  team  mental  models 
via  correlation  analysis,  and  (2)  to  identify  significant  changes  in  team  member  workload  ratings. 
The  similarity  ratings  of  team  member  sources  of  workload  suggest  the  formation  of 
complementary  mental  models  when  information  is  distributed  in  a  complementary  fashion. 
Additionally,  only  the  navigator  experienced  the  higher  workload  changes;  recall  that  the 
navigator  wore  foggles,  an  artificial  constraint  used  only  for  experimental  purposes  in  the 
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complementary  configuration.  This  is  a  dramatic  change  from  normal  operations;  it  was 
anticipated  that  they  would  initially  experience  a  higher  workload.  These  results  indicate  that 
there  is  little  increase  in  workload  to  team  members  under  complementary  conditions,  and  those 
increases  will  decrease  over  time. 


Summary  of  Conclusions 

This  study  has  introduced  a  new  theoretical  concept  of  complementary  team  mental  models 
and  empirical  evidence  has  been  provided  that  supports: 

1 .  The  use  of  a  new  system  design  technique  to  enhance  these  models  through  the 
complementary  distribution  of  information  within  a  team  environment  tailored  to  the 
domain  and  task,  and 

2.  The  benefits  of  complementary  team  mental  models  to  team  process  and  performance. 

The  method  of  designing  team-centered  environments  introduced  in  this  study  is  not  a 
replacement  for  current  methods  used  to  enhance  team  process  and  performance  (e.g.,  cross 
training).  On  the  contrary  it  will  enhance  the  effects  of  current  methods.  The  concepts 
introduced  are  applicable  to  all  models  of  team  performance  but  are  best  incorporated  into  the 
framework  of  the  I-P-O  model  introduced  in  the  background  and  literature  review;  as  shown  in 
Figure  46.  In  the  expanded  I-P-O  model,  characteristics  of  personnel  (position,  pre-existing 
knowledge),  tasks  (team  and  individual),  and  tools  (artifacts  specific  to  the  domain)  are  inputs 
into  the  environment.  Inputs  are  “filtered”  through  the  system  design  of  the  environment. 
Consequently,  the  complementary  distribution  of  information  sources  aligns  information  sources 
within  the  environment  to  be  consistent  with  those  input  characteristics  and  team  processes;  e.g., 
roles  for  each  team  member.  The  information  immediately  available  to  team  members  supports 
their  individual  tasks  and  team  task  responsibilities,  as  required  by  their  role.  Consequently, 
during  nominal  operations,  team  members  can  focus  on  their  individual  tasks  and  are  compelled 
to  explicitly  share  information  with  their  team  member;  during  off-nominal  conditions,  team 
members  know  who  has  access  to  what  information  and  what  information  is  critical  to  share. 

The  interpersonal  interactions  that  are  encouraged  by  a  complementary  information  distribution 
lead  directly  to  the  development  of  complementary  team  mental  models.  As  a  result,  team 
processes  such  as  communication,  role  clarification,  and  decision-making  improve.  The  dashed 
arrow  (Figure  43)  from  the  information  sources  indicates  that  there  may  be  some  direct  impact  on 
team  process  due  to  the  configuration  of  information  sources,  also  impacting  team  performance. 
For  instance,  roles  can  be  clarified  when  an  information  distribution  limits  the  ability  of  one  team 
member  to  perform  tasks  outside  of  their  area  of  responsibility,  or  withholds  critical  decision¬ 
making  information  from  certain  team  members. 
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Figure  43.  Expanded  IPO  framework  with  inclusion  of  complementary  information 
distribution. 


Domains  discussed  in  this  research  frequently  rotate  team  members.  For  example,  aircrews 
in  commercial  and  military  aircraft  very  seldom  stay  together  as  a  team  and  “newly  formed 
teams”  are  the  standard.  The  interchanging  of  team  members  frequently  leads  to  uncertainty  in 
roles  and  performance  expectations.  A  complementary  type  of  system  design  can  clarify  their 
roles  and  promote  explicit  communication  for  teams  that  work  together  infrequently. 

While  improved  outcomes  are  probable,  but  not  guaranteed,  other  external  factors  could  affect 
the  outcome  in  a  positive  or  negative  manner.  In  this  research,  simulator  operators  controlled  the 
external  factors  and  improved  performance  was  generally  experienced  in  both  domains. 

Referring  back  to  Figure  43,  the  “outcomes”  feed  back  into  the  complementary  team  mental 
models  and  information  sources.  Once  performance  (or  other  outcomes  such  as  member  needs  or 
group  maintenance  not  examined  in  this  research)  is  assessed,  the  information  sources  may  need 
to  be  redesigned  and/or  the  complementary  team  mental  models  be  further  developed. 

Complementary  team  mental  models  are  influenced  by  the  distribution  of  knowledge 
between  team  members,  the  distribution  of  information  in  the  environment,  or  a  combination  of 
the  two.  The  quadrants  in  Figure  44  represent  the  levels  of  knowledge  and  information 
distribution  as  described  below: 

•  Quadrant  I:  Team  member  knowledge  is  distributed  among  team  members.  The  information 
in  the  environment  is  common  to  all  team  members.  Based  on  the  division  of  knowledge, 
team  members  are  more  aware  of  their  specific  information  requirements  and  can  identify 
their  required  information  sources  in  the  environment  more  readily. 
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•  Quadrant  II:  Team  member  knowledge  is  distributed  among  team  members  and  the 
information  in  the  environment  is  complementarily  distributed.  The  distribution  of 
information  sources  in  the  environment  supports  the  division  of  knowledge  between  team 
members  and  performance  is  further  enhanced. 

•  Quadrant  III:  Team  knowledge  and  information  are  common  to  all  team  members.  Team 
members  must  rely  heavily  on  training,  operating  procedures,  checklists  and  regulations  to 
align  information  sources  of  the  environment  with  individual  tasks  and  team  responsibilities. 

•  Quadrant  IV:  Team  knowledge  is  common  among  members,  yet  the  information  in  the 
environment  is  complementarily  distributed  between  team  members  according  to  tasks  and 
responsibilities. 


Figure  44.  Knowledge  distribution  vs.  information  distribution. 


While  this  study  only  examined  quadrants  HI  and  IV  in  detail,  these  arguments  suggest  that 
complementary  team  mental  models  are  enhanced  in  quadrants  I,  n,  and  IV,  while  quadrant  III  is 
not  conducive  to  team  operations. 

Additionally,  it  is  hypothesized  that,  as  team  members  increase  repetitions  in  the 
environment,  their  mental  models  will  further  improve  based  on  this  feedback  loop.  Therefore, 
outcomes  are  not  only  a  byproduct  of  the  team  process;  they  are  also  an  input  to  the  system 
design  and  the  development  of  complementary  team  mental  models. 
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Contributions 


Although  this  effort  focused  on  two  specific  domains,  teams  are  found  in  a  variety  of  other 
domains;  examples  include  sports  (NASCAR  pit  crews,  football),  performing  arts  (jazz  band, 
Broadway  show),  emergency  medical  teams,  commercial  aviation  (cockpit  teams,  air  traffic 
control),  and  numerous  others.  This  research  examined  two  sets  of  teams  and  conditions  that 
spanned  a  wide  range  of  possible  teams  within  complex  environments:  (1)  newly  formed,  novice 
teams  performing  a  complex  task  in  a  medium  fidelity  simulator  with  little  to  no  expectation  of 
future  interaction  in  a  team  task;  and  (2)  newly  formed,  highly  trained  U.S.  military  aircrews 
performing  a  complex  task  in  a  high  fidelity  simulator  with  expectations  of  future  interaction  in  a 
team  task.  Generally,  results  concerning  the  communication,  workload,  performance  and  team  of 
mental  models  were  consistent  across  domains,  indicating  high  generalizability  of  these  results 
across  teams  such  as  commercial  airline  crews,  military  aircrews,  and  ground  vehicle  teams,  and 
numerous  other  small  teams  operating  in  complex  domains. 

A  homogeneous  distribution  of  information,  where  all  information  is  available  to  each  team 
member,  would  be  acceptable  if  the  task  responsibilities  were  also  homogeneous;  in  such  a 
situation  a  team  is  most  likely  not  warranted.  Instead,  the  robustness  of  the  information 
distribution  design  principles  presented  in  this  study  is  that  they  account  for  the  heterogeneity  of 
teams;  teams  cannot  be  thought  of  as  groups  of  identical  individuals.  Each  team  member  brings 
his  or  her  own  experiences,  characteristics,  talents  and  own  knowledge  base  to  the  team 
environment.  Team  members  will  apply  their  knowledge  to  interpret  information  about  the 
situation  and  decide  how  to  handle  it  (Wigg,  1998).  The  sources  of  information  within  the 
environments  consist  of  facts  and  data  organized  to  describe  a  particular  situation  or  condition. 
This  study  has  demonstrated  that,  with  newly  formed  teams,  tailoring  information  distribution  in 
accordance  with  team  responsibilities  and  individual  task  requirements  improves  their  team 
processes  and  performance  through  the  development  of  complementary  team  mental  models. 

A  unique  and  generalizable  method  to  determine  information  sources  and  requirements 
within  a  domain  for  a  given  task  has  been  introduced,  demonstrated,  and  forms  the  foundation  of 
complementary  information  distribution.  This  study  re-defines  the  gauge  of  efficiency  for  team 
mental  models  as  “complementariness”  where  each  team  member  (1)  has  the  knowledge 
necessary  to  conduct  his/her  tasks,  (2)  knows  which  information  is  available  to  the  other  team 
member  should  he/she  need  to  seek  it,  (3)  knows  which  information  is  needed  from  them  to  other 
team  members  and  when.  Furthermore,  an  approach  to  information  distribution  that  aligns 
information  sources  with  input  characteristics  and  task  has  been  described  with  empirical 
evidence  that  this  method  can  (1)  reinforce  the  development  of  complementary  team  mental 
models,  and  (2)  improve  team  process  and  performance  without  significantly  affecting  the  team’s 
workload. 


Additional  research 


There  are  improvements  that  could  be  incorporated  in  the  experiments  in  this  research; 
additionally,  interesting  new  avenues  of  research  are  inspired  by  these  results. 
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•  The  focus  of  the  analysis  of  communications  investigated  the  differences  between 
complementary  and  normal  configurations.  Broader  research  into  communication  patterns 
between  levels  of  complementariness  should  be  pursued.  This  research  could  support 
existing  research  concerning  communication  patterns  of  teams  exposed  to  off-nominal  versus 
nominal  conditions. 

•  Communications  were  analyzed  at  the  team  level  for  this  research.  A  more  specific  analysis 
should  focus  on  communications  by  role  (e.g.,  pilot  versus  navigator). 

•  The  method  for  determining  information  sources  and  requirements  should  be  investigated  for 
use  in  other  system  design  processes. 

•  The  effect  of  complementary  information  distribution  during  phased  training  should  be 
investigated.  Even  if  it  is  not  possible  or  not  desirable  to  redesign  a  system,  this  information 
distribution  can  be  used  in  a  training  environment  to  identify  important  elements  of  team 
operations  in  a  specific  domain. 

•  The  outcomes  of  the  helicopter  experiment  revealed  an  error  in  the  information  distribution. 
This  experiment  warrants  redesign  and  further  trials,  which  may  strengthen  conclusions 
concerning  the  task  relevant  anticipation  ratio,  and  continued  research  should  be  conducted  to 
validate  these  results  in  alternate  domains. 

•  Additional  experimentation  should  be  conducted  with  enhanced  measures  of  individual 
mental  models  to  measure  the  direct  effect  of  complementary  information  design  on 
individuals. 

•  Expansion  of  the  concepts  of  information  distribution  to  larger  teams  and  distributed  teams 
should  be  conducted. 
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•This  is  an  urban  environment. 

•The  streets  running  east/west  are  numbered  streets. 

•The  streets  running  north/south  are  lettered  streets. 

•Speed  limit  is  35  MPH  unless  otherwise  posted. 

•Map  is  oriented  north  as  indicated  in  the  upper  right  comer 
of  the  map. 

•The  route  will  be  indicated  by  a  solid  line  as  indicated  on 
the  map. 

•All  roads,  driveways,  parking  lots  etcetera  can  be  driven  on. 
•The  road  at  the  top  of  the  map  in  the  center  is  unusable. 

•Distinctive  buildings  and  areas  are  labeled  (i.e.,  hospital, 
apartments). 

•  Other,  non-distinctive  buildings  will  repeat  throughout  the 
scenario  and  would  not  be  effective  landmarks. 

•There  are  streetlights,  street  signs,  and  one-way  signs 
resembling  those  on  a  normal  city  street. 

•There  may  be  other  vehicles  and  pedestrians  that  are  not 
depicted  on  the  map. 

•Circle  indicates  Start  Point,  Square  indicates  End  Point. 


Sample  Textual  Directions 

Starting  Point:  Hospital  parking  lot. 

•Exit  Parking  lot  and  turn  east  on  1  st  Street. 

•Check  Point:  Pass  DMV  on  your  left. 

•At  second  intersection  turn,  turn  North  onto  E  Street. 
•Turn  West  on  2nd  Street. 

•Check  Point:  There  will  be  a  gas  station  and  a  fire  station 
on  your  right. 

•When  you  reach  the  fire  station  turn  North  on  D  Street. 
•Take  your  turn  East  onto  5th  Street  and  travel  one  block. 


•Turn  South  on  E  Street  (School  Zone:  Speed  Limit  drops 
to  25  Mph). 


•Turn  east  into  the  school  parking  lot. 
Destination:  School  narkinn  lot. 
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Introductory  brief 


During  this  experiment,  you  will  be  operating  an  automobile  simulator.  You  will  be  asked 
to  participate  in  a  total  of  six  driving  scenarios:  one  training  scenario  and  five  experimental  runs. 

The  training  scenario  is  designed  to  familiarize  you  with  the  controls  of  the  simulator  and  the 
type  of  scenario  that  you  will  operate  in.  The  experimental  runs  are  scenarios  during  which  team 
performance  and  behavior  will  be  observed  and  recorded.  In  each  experimental  run,  you  will  be 
asked  to  drive  to  a  specified  location  in  the  city;  please  do  your  best  to  obey  all  traffic  laws  (i.e., 
speed  limit  [55  unless  otherwise  posted ],  traffic  signals,  stop  signs,  turn  signal  usage,  etc.).  This 
is  a  team  operation,  and  you  are  both  responsible  for  these  tasks. 

This  experiment  will  last  approximately  1. 5-2.0  hours.  At  the  beginning  of  each  scenario, 
the  navigator  and  sometimes  the  driver  will  be  given  a  sheet  with  the  textual  directions  and 
navigation  map  with  a  highlighted  route  (you  will  have  one  minute  to  review  the  route).  The 
simulation  starts  when  the  driver  turns  the  ignition  key  and  ends  when  the  vehicle  is  shut  down. 
Please  do  not  discuss  the  scenario  with  your  teammate  until  the  simulation  is  started.  You  will 
leave  from  the  Starting  Point  (SP)  and  should  attempt  to  stay  on  the  given  route  as  best  as 
possible.  If  you  deviate  from  the  route,  please  attempt  to  get  back  on  the  route  in  the  most 
expedient  manner.  There  will  be  other  cars  on  the  road  in  each  scenario.  During  the 
experimental  runs  please  refrain  from  giving  hand  signals,  gestures  or  physically  pointing  to 
objects;  the  only  form  of  communication  you  should  use  is  verbal  communication. 

You  will  be  provided  various  breaks  during  the  course  of  the  experiment;  furthermore,  you 
may  request  a  break  at  any  time  that  you  desire.  We  ask,  if  possible,  that  you  avoid  requesting  a 
break  during  the  periods  when  you  are  actually  operating  the  simulator.  However,  if  you  are 
feeling  sick  during  the  simulation  or  need  to  stop  it  for  any  reason,  please  let  us  know  and  we 
will  stop  the  simulation  immediately. 

Throughout  the  course  of  the  experiment,  feel  free  to  verbalize  your  thought  process.  Each 
run  will  be  video  taped  for  further  review.  During  this  experiment,  we  plan  to  examine  team 
performance  while  you  are  conducting  a  navigation  task  under  a  time  constraint.  Please  do  your 
best  to  act  naturally  and  drive  the  vehicle  in  the  same  manner  in  which  you  would  drive  your  own 
vehicle.  We  would  like  to  get  the  best  estimate  of  a  ‘real-life’  response.  This  will  help  us 
ascertain  the  consequences  of  different  information  structures  on  team  performance. 

Please  help  yourself  to  something  to  drink  if  you  would  like.  We  also  have  Internet 
available  on  various  computers  should  you  need  to  access  your  e-mail  or  the  World  Wide  Web 
during  the  course  of  the  experiment. 
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Automobile  simulator  description 


GE  Capital  I-Sim's  PatrolSim™  is  a  compact,  high-performance  driving  simulator  for  the 
law  enforcement  and  government  marketplace.  It  utilizes  GE  Capital  I-Sim's  state-of-the-art 
simulation  technology,  which  provides  a  highly  realistic  and  immersive  training  environment. 
The  PatrolSim  offers  more  than  60  specific  scenarios  for  law  enforcement  driver  training. 

The  PatrolSim  driving  simulator  provides  an  open-seat  driving  station  in  a  low-cost,  high 
fidelity  driving  environment  that  is  suitable  for  training  and  research  applications.  Its  Operator 
Console  provides  interactive,  real-time  control  of  the  driving  environment.  PatrolSim 
incorporates  GE  Capital  I-Sim's  proprietary  vehicle  dynamics,  traffic  scenario  and  road  surface 
software  to  provide  accurate  stimuli  for  the  driver. 

PatrolSim  is  a  stand-alone,  expandable  and  upgradeable  system  capable  of  simulating 
conditions  for  police  vehicles,  fire  trucks,  ambulances,  pickups  and  public  works  vehicles. 
Highlights  include: 

o  Open-seat  driving  configuration  with  typical  police  cruiser  dash,  including  all 
instrumentation  and  controls. 

o  Expanded  horizontal  field  of  view  (FOV)  provides  up  to  270  degrees. 

o  Advanced  graphics. 

o  Powerful  scenario-creation  editing  tools  allow  instructors  to  quickly  create  and 
edit  scenarios  and  run  them  within  minutes. 

o  Scenario  traffic  incorporates  artificial  intelligence;  vehicles  behave  as  in  real  life 
or  can  be  controlled  in  real  time. 

o  Open  architecture  Windows™-based  software  running  on  off-the-shelf 
commercially  available  PC  computer  platforms. 
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End  of  run  questionnaire  -  NASA  task  load  index  (TLX) 

We  are  interested  not  only  in  assessing  your  performance  but  also  your  experiences  in  the 
different  conditions.  Basically  I  want  to  examine  your  "workload." 

Since  workload  is  something  experienced  individually,  it  can  be  difficult  to  estimate.  Because 
workload  may  be  influenced  by  many  different  factors,  we  would  like  you  to  evaluate  several 
factors  individually  rather  than  lumping  them  into  a  single  evaluation  of  overall  workload.  This 
set  of  six  rating  scales  was  developed  by  NASA.  Please  read  the  descriptions  of  the  scales 
carefully.  If  you  have  a  question  about  any  of  the  scales,  please  ask  us  about  it,  as  it  is  important 
that  they  be  clear  to  you.  I  will  leave  the  descriptions  on  the  table  for  reference  during  the  rest  of 
the  experiment. 

Please  evaluate  the  scenario  by  marking  each  scale  at  the  point  that  matches  your  experience. 
Each  line  has  two  endpoint  descriptors  that  describe  the  scale.  Note  that  "performance"  goes 
from  "good"  on  the  left  to  "bad"  on  the  right.  Please  place  an  X  anywhere  along  each  scale 
between  a  pair  of  tick  marks.  Consider  each  scale  individually.  These  ratings  are  an  important 
part  of  the  experiment  and  I  appreciate  your  efforts. 
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Rating  scale  definitions 


Title 

MENTAL 

DEMAND 


PHYSICAL 

DEMAND 


TEMPORAL 

DEMAND 


EFFORT 


PERFORMANCE 


FRUSTRATION 

LEVEL 


Descriptions 

How  much  mental  and  perceptual  activity  was  required  (e.g., 
thinking,  deciding,  calculating,  remembering,  looking, 
searching,  etc.)?  Was  the  task  easy  or  demanding,  simple  or 
complex,  exacting  or  forgiving? 

How  much  physical  activity  was  required  (e.g.,  pushing, 
pulling,  turning,  controlling,  activating,  etc.)?  Was  the  task 
easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful 
or  laborious? 

How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace 
at  which  the  tasks  or  task  elements  occurred?  Was  the  pace 
slow  and  leisurely  or  rapid  and  frantic? 

How  hard  did  you  have  to  work  (mentally  and  physically)  to 
accomplish  your  level  of  performance? 

How  successful  do  you  think  you  were  in  accomplishing  the 
goals  of  the  task  set  by  the  experimenter  (or  yourself)?  How 
satisfied  were  you  with  your  performance  in  accomplishing 
these  goals? 

How  insecure,  discouraged,  irritated,  stressed  and  annoyed 
versus  secure,  gratified,  content,  relaxed  and  complacent  did 
you  feel  during  the  task? 


105 


Position  (Driver  or  Navigator). 

Team  Number _ 

Information  Level _ 

Scenario _ 

Run  order _ 

Date _ 


Mental  Demand 


Low  High 


Physical  Demand 

i  I  i  I  i  I  i  I  i  I  i  I  i  I  i  I  i  I  i 

Low  High 

Temporal  Demand 

i  I  i  1  i  I  i  1  i  I  i  I  i  I  i  I  i  I  i 

Low  High 

Effort 

I  I  1  1  1 .  1  1  1  1  1  1  1  I  1  I  1  I  1  1 

Low  High 

Performance 

-J  1  I  I  I  1  1  I  I  1  I  1  1  I  I  1  I  I  I  1 

Good  Poor 


Frustration 


Low  High 


For  the  following  questions,  please  focus  on  the  interaction  between  you 
and  your  teammate,  not  the  difficulty  due  to  road  conditions,  etc. 

What  did  you  find  most  difficult  about  this  scenario?  Why? 


What  did  you  find  easiest  about  this  scenario?  Why? 
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End-of-experiment  questionnaire 


Position  (Driver  or  Navigator) _ 

Team  Number _ 

Date _ 

Age: _ 

Male  or  Female  (circle  one) 

Years  with  a  valid  driver's  license: _ 

1)  How  familiar  are  you  with  the  team  member  with  whom  you  participated  in  the  experiment? 

1  2  3  4  5 

(Unfamiliar:  just  met)  (Very  familiar:  communicate 

with  them  daily) 

2)  Have  you  ever  performed  coordinated  tasks  with  this  team  member? 

1  2  3  4  5 

(Never)  (Frequently:  work  with 

them  daily) 

3)  How  difficult  was  it  to  coordinate  your  actions  with  your  teammate?  (Circle  number) 


Scenario 

A:  Both  driver  and  navigator  had 
maps/directions 
B:  Only  the  navigator  had 
map/directions 

C:  Both  driver  and  navigator  had 
maps/directions.  The  navigator  wore  eye 
devices  limiting  vision. 

D:  Only  the  navigator  had 
map/directions.  The  navigator  wore  eye 
devices  limiting  vision. 


1=Very  4=Very 

Difficult  2=Difficult  3=Easy  Easy 

Difficulty  Level 


1  2  3  4 

1  2  3  4 

1  2  3  4 

1  2  3  4 


Please  address  any  reasons  why  you  believe  this  was  so: 


4)  Have  you  ever  had  any  type  of  formal  training  on  teamwork  (i.e.,  sports,  crew  resource 
management,  a  short  course  etc.)? 


1  2  3  4  5  None 


(Informal) 


(Formal:  classroom 
instruction) 
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If  so,  please  specify  what  type  of  training. 


5)  Which  form  of  directions  did  you  find  most  useful  during  normal  navigating:  textual,  map 
with  route,  or  a  combination?  Explain. 


6)  Which  form  of  directions  did  you  find  most  useful  during  abnormal  periods  of  navigating 
(e.g..,  getting  back  on  your  route  after  a  wrong  turn):  textual,  map  with  route,  or  a  combination? 
Explain. 


7)  Regarding  communication  techniques  used  by  your  teammate,  what  would  you  like  him  or  her 
to  continue  to  do  in  future  scenarios,  improve  upon,  or  do  differently? 


8)  Regarding  communication  techniques  used  by  you,  what  would  you  like  to  continue  to  do  in 
future  scenarios,  improve  upon,  or  do  differently? 


108 


Scenario  maps 


Figure  A-2.  Scenario  1  map. 


109 


Scenario  1  Textual  Directions 


Starting  Point:  Hospital  parking  lot. 

•Exit  Parking  lot  and  turn  East  on  1st  Ave. 

•Pass  DMV  on  your  left. 

•At  second  left  turn,  just  past  police  HQ,  turn  North  onto  E  Street  (this  is  a  one  way 
street). 

•Turn  West  on  2nd  Ave. 

•There  will  be  a  gas  station  and  a  fire  station  on  your  right. 

•When  you  reach  the  fire  station  turn  North  on  D  Street. 

•After  one  block  turn  West  on  3rd  Ave. 

•Turn  North  on  B  Street. 

•At  your  second  intersection  turn  West  on  5th  Ave. 

•Turn  North  on  A  Street. 

•At  your  second  intersection  turn  East  on  9th  Ave. 

•Turn  North  on  B  Street. 

•You  will  come  to  a  “T”  intersection,  at  this  intersection  turn  East  on  10th  Ave. 
•Travel  on  10th  Ave  for  three  blocks. 

•At  your  third  intersection  turn  South  on  E  Street. 

•Travel  South  on  E  Street  for  approximately  .3  miles. 

•At  your  third  intersection  turn  East  on  6th  Ave. 

•Turn  South  on  F  Street. 

•At  your  first  intersection  turn  East  on  5th  Ave. 

•Turn  North  on  G  Street  and  make  your  first  turn  East  on  6th  Ave. 

•You  will  come  to  a  “T”  intersection,  at  this  intersection  turn  South  on  H  Street. 
•Turn  West  on  3rd  Ave. 

•Turn  South  on  G  Street. 

•Turn  West  on  2nd  Ave. 

•At  your  first  intersection  turn  South  on  F  Street  for  one  block. 

•You  will  come  to  a  “T”  intersection,  at  this  intersection  turn  East  on  1st  Ave. 

•Turn  north  into  the  Mall  parking  lot. 

•Drive  up  to  the  entrance  of  Dills. 

Destination:  Mall  parking  lot  at  the  entrance  of  Dills. 
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Do  Not  Distribute 


Figure  A-3.  Scenario  2  map 


Scenario  2  Textual  Directions 


Starting  Point:  Facing  East  at  the  intersection  of  E  Street  and  9th  Ave. 

•Turn  South  on  E  Street. 

•Turn  West  on  7th  Ave  and  travel  one  block. 

•Turn  South  on  D  Street,  D  St.  will  intersect  with  6th  Ave,  continue  on  D  Street  to  the 
East  and  South. 

•Turn  West  on  5th  Ave  for  one  block. 

•Turn  South  on  C  Street. 

•Turn  West  on  4th  Ave. 

•Turn  North  on  B  Street  for  three  blocks. 

•At  your  third  intersection  turn  West  on  8th/Green. 

•Turn  West  on  7th  Ave. 

•Turn  South  on  A  Street  for  approximately  .3  miles. 

•Turn  East  on  2nd  Ave. 

•Travel  on  2nd  Ave  for  approximately  lA  of  a  mile. 

•Turn  South  on  D  Street. 

•You  will  come  to  a  “T”  intersection,  at  this  intersection  turn  East  on  1 S1  Ave,  Police  HQ 
will  be  on  your  left  and  DMV  will  be  on  your  right. 

•Travel  East  on  1st  Ave  for  approximately  .3  miles. 

•Turn  North  on  H  Street. 

•Travel  on  H  Street  for  approximately  .2  miles. 

•Turn  West  on  3rd  Ave  for  one  block. 

•Turn  North  on  G  Street. 

•Turn  West  on  5th  Ave. 

•At  the  first  intersection  turn  South  on  F  Street. 

•Turn  West  on  2nd  Ave. 

•At  you  second  intersection  turn  North  on  E  Street  (This  is  a  one  way  street). 

•Turn  East  into  the  school  Parking  lot  and  pull  up  to  the  front  entrance. 

Destination:  School  parking  lot  at  the  entrance. 
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Do  Not  Distribute 


Figure  A-4.  Scenario  3  map. 


Scenario  3  Textual  Directions 


Starting  Point:  Mall  parking  lot  at  the  entrance  of  Dills. 

•Exit  the  mall  parking  lot  and  turn  East  onto  1st  Avenue. 

•Travel  one  block  and  turn  North  on  H  Street. 

•Turn  West  on  3rd  Avenue. 

•At  the  first  intersection  turn  North  on  G  Street. 

•Turn  East  on  5th  Avenue. 

•You  will  come  to  a  T-intersection,  turn  North  on  H  Street. 

•Take  your  first  turn  West  on  6th  Avenue. 

•Travel  on  6th  Avenue  for  three  blocks,  at  the  third  intersection  turn  North  on  E 
Street. 

•At  your  third  four  way  intersection  turn  West  on  10th  Avenue. 

•Turn  South  on  B  Street. 

•Turn  West  on  9th  Avenue. 

•You  will  come  to  a  T-intersection,  turn  South  onto  A  Street. 

•At  the  second  intersection  turn  East  on  5th  Avenue. 

•Turn  South  at  the  first  intersection  on  B  Street. 

•At  the  first  intersection  turn  East  on  4th  Avenue. 

•Take  your  first  turn  South  onto  C  Street. 

•At  your  first  intersection  turn  West  on  3rd  Avenue. 

•Turn  South  on  B  Street. 

•You  will  come  to  a  T-intersection,  at  this  intersection  turn  West  on  2nd 
Avenue. 

•  Turn  South  on  A  Street. 

•Turn  East  on  1st  Avenue,  the  hospital  will  be  on  your  left. 

•Drive  past  the  Department  of  Motor  Vehicles. 

•Turn  into  the  Police  Department  Head  Quarters. 


Destination:  Police  Department. 
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Scenario  4  Textual  Directions 


Starting  Point:  Facing  East  at  the  bend  in  the  road  on  B  Street. 

•Follow  B  Street  East  and  South  until  you  reach  9th  Ave. 

•Turn  East  on  9th  Ave  and  travel  for  two  blocks 

•At  your  second  intersection  turn  North  on  D  Street 

•You  will  come  to  a  T-intersection,  turn  East  at  this  intersection  onto  10th 

Ave 

•At  your  second  intersection  turn  South  on  F  Street 
•At  your  third  intersection  turn  East  on  6th  Ave 
•Turn  South  on  H  Street 
•Take  your  first  intersection  West  on  5th  Ave 
•Turn  North  on  E  Street 

•At  your  first  four  way  intersection  turn  West  on  7th  Ave 
•You  will  come  to  a  T-intersection,  turn  South  on  A  Street. 

•Travel  one  block  on  A  Street  and  turn  East  on  5th  Ave 

•At  your  first  intersection  turn  North  on  B  Street 

•Turn  East  on  6h  Ave 

•Turn  South  on  D  Street 

•At  your  first  intersection  turn  West  on  5th  Ave 

•Turn  South  on  C  Street 

•Turn  West  on  4th  Ave 

•Turn  South  on  B  Street 

•At  your  first  four  way  intersection  turn  East  on  3rd  Ave 
•At  your  second  intersection  turn  South  on  D  Street 
•You  will  come  to  a  T-intersection,  turn  West  on  1st  Street. 

•Turn  North  into  the  parking  lot  of  the  Department  of  Motor  Vehicles  and 
park  near  the  front  entrance 

Destination:  Department  of  Motor  Vehicles 
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Figure  A-6.  Scenario  5  map. 
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Scenario  5  Textual  Directions 


Starting  Point:  Facing  West  at  the  intersection  of  5th  Ave  and  F  Street. 


•Travel  West  approximately  .3  miles  on  5th  Ave. 

•Turn  South  at  your  fourth  intersection  on  to  C  Street. 

•At  your  second  intersection  turn  West  on  3rd  Ave. 

•At  your  1st  intersection  turn  North  on  B  Street. 

•At  your  second  intersection  turn  West  on  5th  Ave. 

•You  will  come  to  a  “T”  intersection,  turn  North  on  A  Street. 
•Turn  East  on  7th  Ave. 

•Make  an  immediate  turn  North  on  Green. 

•Follow  this  to  B  Street  and  Turn  North  on  B  Street. 

•You  will  come  to  a  “T”  intersection,  turn  East  on  10th  Ave. 
•At  your  second  intersection  turn  South  on  D  Street. 

•Follow  D  Street  all  the  way  to  2nd  Ave. 

•Turn  East  on  1st  Ave. 

•Turn  North  on  E  Street. 

•Turn  West  into  the  Hotel  parking  lot. 

•Drive  up  to  the  entrance,  under  the  overhang. 

Destination:  Hotel  parking  lot. 
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Automobile  participant  effects  (p-Values) 


Table  A-l. 

Automobile  participant  effects  on  communications. 


W 

G) 

3 

CT 

0) 

c 

12 

a> 


I- 

(0 

<*— 

c 

0) 

E 

G> 

D1 

TJ 

0) 

5 

o 

c 


Total 
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Action 
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Action 
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Specific 
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Comms 

Ratio 


Age 

Gender 

Mix 

Experience 

Experience 

Spread 

Familiarity 

Coordinated 

Task 

Experience 

Teamwork 

Training 

Training 

(Run) 

0.31 

0.10 

0.40 

0.31 

0.60 

0.62 

0.60 

0.07 

0.49 

0.08 

0.40 

0.49 

0.62 

0.17 

0.40 

0.40 

0.33 

0.09 

0.42 

0.33 

0.56 

0.63 

0.62 

0.08 

0.88 

0.08 

0.75 

0.88 

0.13 

0.20 

0.16 

0.95 

0.00 

0.66 

0.24 

0.00 

0.13 

0.12 

0.20 

0.85 

0.33 

0.00 

0.90 

0.33 

0.48 

0.59 

0.70 

0.85 

0.56 

0.22 

0.44 

0.56 

0.94 
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0.00 
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0.00 

0.22 

0.22 

0.47 

0.99 

0.12 

0.81 

0.00 

0.12 

0.14 

0.07 

0.12 

0.36 

0.87 

0.39 

0.18 

0.87 

0.02 

0.02 

0.00 

0.87 

0.41 

0.55 

0.30 

0.41 

0.48 

0.69 

0.10 

0.35 

0.01 

0.25 

0.38 

0.01 

0.20 

0.28 

0.98 

0.89 

0.00 

0.67 

0.39 

0.00 

0.83 

0.62 

0.31 

0.94 

Table  A-2. 

Automobile  participant  effects  on  performance. 


Training 

Age 

Gender  Mix 
Experience 
Experience  Spread 
Familiarity 

Coord.  Task  Experience 
Teamwork  Training 
Order  1 
Order  2 


iilljSli 

mu 

t  r:  1 1* 

X 

X 

0.23 

0.75 

0.51 

0.14 

0.93 

0.48 

0.68 

0.44 

0.60 

0.07 

0.32 

0.69 

0.44 

liW 

0.58 

0.15 

i  0.44 

0.24 

0.33 

0.08 

0.51 

0.02 

0.13 

0.59 

0.20 

0.36 

0.48 

0.44 

0.60 

0.07 

0.32 

0.69 

0.72 

0.46 

0.15 

0.13 

0.11 

0.49 

0.53 

0.64 

0.68 

0.11 

0.16 

0.14 

0.67 

0.72 

0.26 

0.39 

0.12 

0.19 

0.57 

0.41 

X 

mm 

0.25 

0.33 

0.78 

0.38 

0.32 

X 

mm 

0.95 

0.73 

0.37 

0.28 

0.23 

Order  1 :  #  runs  prior  to  complementary  run 
Order  2:  #  of  previous  complementary  runs 
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Appendix  B. 


Experiment  II:  background  documentation. 

Introductory  Briefing  For  Participants 

During  this  experiment  you  will  be  operating  the  NUH-60  Black  Hawk  full  motion 
simulator.  You  will  be  asked  to  participate  in  a  total  of  four  flight  scenarios:  one  training 
scenario  and  three  experimental  runs.  The  training  scenario  is  designed  to  familiarize  you  with 
the  controls  of  the  simulator  and  the  type  of  scenario  that  you  will  operate  in.  The  experimental 
runs  are  scenarios  during  which  team  performance  and  behavior  will  be  observed  and  recorded. 
In  each  experimental  run,  you  will  be  asked  to  conduct  a  normal  take-off  and  then  navigate  to 
and  land  at  a  specified  location  using  a  standard  route  card  and  Flight  Information  Publications; 
please  do  your  best  to  follow  all  flight  profile  instructions  (i.e.,  airspeed,  altitude,  heading,  climb 
rate,  etc.).  These  are  team  operations  and  you  are  both  responsible  for  these  tasks. 

This  experiment  will  last  approximately  two  to  three  hours.  At  this  time,  we  will  designate 
one  of  you  as  the  pilot  and  one  of  you  as  the  co-pilot/navigator.  You  will  remain  in  these 
positions  throughout  the  experiment.  At  the  beginning  of  each  scenario,  the  navigator,  and 
sometimes  the  pilot,  will  be  given  a  map  with  a  route  posted;  you  will  also  be  given  a  route  card 
with  headings,  altitudes,  airspeeds,  and  checkpoints,  approach  plates  for  local  airfields,  and  a 
description  of  the  landing  area.  The  runs  are  each  approximately  15  minutes  long  and  a  break  is 
given  between  them.  The  collection  of  data  begins  when  you  are  given  the  route  information  and 
ends  when  the  aircraft  has  landed.  Please  do  not  discuss  the  scenario  with  your  teammate  until 
the  simulation  is  started.  You  will  leave  from  the  Starting  Point  (SP)  and  should  attempt  to  stay 
on  the  given  route  as  best  as  possible.  If  you  deviate  from  the  route,  please  attempt  to  get  back 
on  the  route  in  the  most  expedient  manner.  During  the  experimental  runs,  please  refrain  from 
giving  hand  signals,  gestures  or  physically  pointing  to  objects;  the  only  form  of  communication 
you  should  use  is  verbal  communication. 

You  will  be  provided  breaks  during  the  course  of  the  experiment;  furthermore,  you  may 
request  a  break  at  any  time  that  you  desire.  We  also  ask,  if  possible,  that  you  avoid  requesting  a 
break  during  the  periods  when  you  are  actually  operating  the  simulator.  However,  if  you  are 
feeling  sick  during  the  simulation  or  need  to  stop  it  for  any  reason,  please  let  us  know  and  we 
will  stop  the  simulation  immediately. 

Throughout  the  course  of  the  experiment,  feel  free  to  verbalize  your  thought  process.  Each 
run  will  be  video  taped  for  further  review.  During  this  experiment,  we  plan  to  examine  team 
performance  while  you  are  conducting  a  navigation  task  under  a  time  constraint.  Please  do  your 
best  to  act  naturally  and  fly  the  aircraft  in  the  same  manner  in  which  you  would  while  conducting 
a  real  world  mission.  We  would  like  to  get  the  best  estimate  of  a  ‘real-life’  response.  This  will 
help  us  ascertain  the  consequences  of  different  information  structures  on  team  performance. 
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End-of-run  questionnaire  -  NASA  task  load  index  TLX 


We  are  interested  not  only  in  assessing  your  performance  but  also  your  experiences  in  the 
different  conditions.  Basically  I  want  to  examine  your  "workload." 

Since  workload  is  something  experienced  individually,  it  can  be  difficult  to  estimate.  Because 
workload  may  be  influenced  by  many  different  factors,  we  would  like  you  to  evaluate  several 
factors  individually  rather  than  lumping  them  into  a  single  evaluation  of  overall  workload.  This 
set  of  six  rating  scales  was  developed  by  NASA.  Please  read  the  descriptions  of  the  scales 
carefully.  If  you  have  a  question  about  any  of  the  scales,  please  ask  us  about  it,  as  it  is  important 
that  they  be  clear  to  you.  I  will  leave  the  descriptions  on  the  table  for  reference  during  the  rest  of 
the  experiment.  You  will  be  asked  to  complete  two  of  these  rating  scales  for  each  run. 

1.  Individual  Workload  Assessment:  Please  evaluate  the  scenario  by  marking  each  scale  at 
the  point  that  matches  your  experience.  Each  line  has  two  endpoint  descriptors  that 
describe  the  scale.  Note  that  "performance"  goes  from  "good"  on  the  left  to  "bad"  on  the 
right.  Please  place  an  X  anywhere  along  each  scale  between  a  pair  of  tick  marks. 
Consider  each  scale  individually. 

2.  Crew-member  Workload  Assessment:  In  the  same  manner  as  above,  please  evaluate  the 
scenario  by  marking  each  scale  at  the  point  that  matches  your  assessment  of  your  crew¬ 
member’s  experience. 

These  ratings  are  an  important  part  of  the  experiment  and  I  appreciate  your  efforts. 
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Rating  scale  definitions 


Title 

MENTAL 

DEMAND 


PHYSICAL 

DEMAND 


TEMPORAL 

DEMAND 


EFFORT 


PERFORMANCE 


FRUSTRATION 

LEVEL 


Descriptions 

How  much  mental  and  perceptual  activity  was  required 
(e.g.,  thinking,  deciding,  calculating,  remembering, 
looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

How  much  physical  activity  was  required  (e.g.,  pushing, 
pulling,  turning,  controlling,  activating,  etc.)?  Was  the 
task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous, 
restful  or  laborious? 

How  much  time  pressure  did  you  feel  due  to  the  rate  or 
pace  at  which  the  tasks  or  task  elements  occurred?  Was 
the  pace  slow  and  leisurely  or  rapid  and  frantic? 

How  hard  did  you  have  to  work  (mentally  and  physically) 
to  accomplish  your  level  of  performance? 

How  successful  do  you  think  you  were  in  accomplishing 
the  goals  of  the  task  set  by  the  experimenter  (or  yourself)? 
How  satisfied  were  you  with  your  performance  in 
accomplishing  these  goals? 

How  insecure,  discouraged,  irritated,  stressed  and 
annoyed  versus  secure,  gratified,  content,  relaxed  and 
complacent  did  you  feel  during  the  task? 
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Position 

Pilot  /  Naviqator 

Crew  Number 

123456789  10 

Run 

T  1  2  3 

Information 

A  B 

Scenario  (leq) 

T  1  2  3 

Date 

Assessment  of  Your  Workload 


Mental  Demand 


Low  High 


Physical  Demand 


Low 


Temporal  Demand 


Low 

Effort 


Low 

Performance 


Good 


High 


High 


High 


Poor 


Frustration 


Position 

Pilot  /  Navigator 

Crew  Number 

123456789  10 

Run 

T  1  2  3 

Information 

A  B 

Scenario  (leg) 

T  1  2  3 

Date 

Assessment  of  Crewmember’s  Workload 


Mental  Demand 


Low  High 


Physical  Demand 


Low  High 


Temporal  Demand 


Low  High 

Effort 


Low  High 

Performance 


Good  Poor 


Frustration 


Using  the  descriptions  in  the  table  below,  please  rank  importance  of  the  types  of  information 
during  different  phases  of  flight  for  this  particular  flight.  Place  a  ‘1’  next  to  the  information  type 
that  is  the  most  important,  a  ‘2’  next  to  the  information  type  that  is  the  next  most  important  etc., 
until  you  reach  ‘6’  the  least  important: 


Phase  of  Flight 


Information 

Type 

Description 

Current  Flight 
Information 

The  current  airspeed, 
altitude,  heading,  climb  rate 
etc.  of  the  aircraft 

Future  Flight 
Information 

The  future  airspeed, 
altitude,  heading,  climb  rate 
etc.  that  you  will  be 
required  to  fly 

Current 

Environmental 

Information 

Description  of  the 
environment,  terrain, 
checkpoints,  etc.  that  you 
are  currently  flying  in/over 

Future 

Environmental 

Information 

Description  of  the 
environment,  terrain, 
checkpoints,  etc.  that  you 
will  be  flying  in/over  in  the 
near  future 

Aircraft 

Instrumentation 

Status 

Status  of  the  aircraft 
instrumentation  (fuel, 
engine  instruments,  etc) 

Other 

Any  other  type  of 
information  that  was 
mportant  for  mission 
completion 

If  you  used  ‘other’  for  any  phase  of  flight,  please  specify  the  information  below. 
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Position 

Pilot  /  Naviqator 

Crew  Number 

123456789  10 

Run 

T  1  2  3 

Information 

A  B 

Scenario  (leq) 

T  1  2  3 

Date 

•  For  your  crewmember  to  provide  to  you: 


Phase  of  Flight 

Before  Take-off 

During  Take-off 

En-route 

Before  Landing 

During  Landing 

After  Landing 

Information 

Type 

Description 

Current  Flight 
Information 

The  current  airspeed, 
altitude,  heading,  climb  rate 
etc.  of  the  aircraft 

■ 

■ 

■ 

Future  Flight 
Information 

The  future  airspeed,  altitude, 
heading,  climb  rate  etc.  that 
you  will  be  required  to  fly 

Current 

Environmental 

Information 

Description  of  the 
environment,  terrain, 
checkpoints,  etc.  that  you 
are  currently  flying  in/over 

Future 

Environmental 

Information 

Description  of  the 
environment,  terrain, 
checkpoints,  etc.  that  you 
will  be  flying  in/over  in  the 
near  future 

Aircraft 

Instrumentation 

Status 

Status  of  the  aircraft 
instrumentation  (fuel, 
engine  instruments,  etc) 

Other 

Any  other  type  of 
information  that  was 
important  for  mission 
completion 

If  you  used  ‘other’  for  any  phase  of  flight,  please  specify  the  information  below. 


End-of-experiment  questionnaire 


Please  answer  the  following: 

Age:  Male  Female 

Rank: 

Date  of  Flight  School  Graduation  (month/year): 

Total  #  of  Flight  Hours: _  Total  #  of  Simulator  Hours:. 

Total  #  of  Military  Flight  Hours:  FW: _  RW: _ 

Total  #  of  Civilian  Flight  Hours:  FW: _  RW: _ 

Please  list  all  aircraft  in  which  you  are  qualified: 


Please  circle  all  types  of  mission  experience  that  you  have  had: 

Observation/Scout  Attack  Transport  Medevac 

Search  and  Rescue  External  cargo  Drug  Interdiction  Training/Instructor 

Other. _ _ _ _ 


Please  circle  the  most  accurate  responses. 

How  well  do  you  know  your  testing  partner? 

a)  not  at  all,  first  time  meeting 

b)  we’ve  met,  briefly  or  social  occasions 

c)  same  unit  but  do  not  work  together 

d)  same  unit  and  work  together,  but  do  not  fly  together  on  a  regular  basis 

e)  fly  together  frequently  (battle  rostered) 


Approximately  how  many  hours  have  you  flown  with  your  testing  partner  in  the  last  year? 
a)  0  b)  1-25  c)  25-50 

d)  50-75  e)  75-  100  f)  100+ 


Approximately  how  many  hours  have  you  flown  with  your  testing  partner  overall? 
a)  0  b)  1-100  c)  100-200 

d)  200- 300  e)  300-400  f)  400+ 
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How  difficult  was  it  to  coordinate  your  actions  with  your  teammate?  (Circle  number) 


Scenario 

A:  Both  pilot  and  navigator 
had  maps/instructions. 

B:  Only  the  navigator  had 
map/instructions.  The 
navigator  wore  eye  devices 
limiting  vision. 


1=Very  4=Very 

Difficult  2=Difficult  3=Easy  Easy 

Difficulty  Level 


12  3  4 


12  3  4 


Please  address  any  reasons  why  you  believe  this  was  so: 


Which  form  of  directions  did  you  find  most  useful  during  normal  navigating:  textual,  map  with 
route,  or  a  combination?  Explain. 


Which  form  of  directions  did  you  find  most  useful  during  abnormal  periods  of  navigating  (e.g., 
IMC,  EP):  textual,  map  with  route,  or  a  combination?  Explain. 
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Scenarios 


The  following  scenarios  will  be  used  for  the  data  collection  runs. 

Flight  instructions  to  pilots  and  co-pilots  for  all  scenarios. 

a.  Follow  all  airspeed,  altitude  and  ground  track  requirements  of  the  waypoint  cards. 

b.  After  all  takeoffs,  climb  straight  ahead  to  400  feet  AGL  before  turning  to  desired 
heading/course. 

c.  All  climbs  will  be  performed  at  500  feet  per  minute;  descents  will  be  performed  as 
necessary. 

d.  All  instrument  turns  will  be  performed  at  standard  rate  (3°  per  second). 

e.  Perform  all  radio  communications  (calls  to  Ground  and  Air  Traffic  Controllers, 
tactical  sites.  Approach  Controllers,  etc.)  that  would  be  required  during  any  actual 
mission. 

f.  Force  trim  will  be  disabled. 
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Scenario  A:  nominal  condition  run 


The  simulator  operator  will: 

1.  Set  visibility  to  3  statute  miles,  ceilings  to  1500  feet,  winds  and  turbulence  to  zero. 

2.  Set  time  of  day  at  12  noon. 

3.  Disable  the  GPS. 

Pre-mission  briefing: 

You  are  assigned  the  mission  of  transporting  two  passengers  (captured  A1  Qaeda  operatives) 
from  Campbell  Army  Airfield  to  FARP  1.  The  mission  is  critical  and  must  be  completed  to  avoid 
an  international  incident.  The  only  reason  to  abort  or  modify  the  mission  will  be  in  the  event  of 
an  emergency  situation  requiring  a  landing  as  soon  as  possible.  Due  to  security  concerns,  a 
special  VMC  flight  corridor  to  the  FARP  has  been  established.  The  GPS  is  inoperative  and  you 
must  navigate  by  pilotage  and  dead  reckoning.  You  must  follow  altitude,  airspeed  and  ground 
track  restrictions  in  accordance  with  the  waypoint  card  in  order  to  comply  with  the  corridor 
requirements.  Maintain  visual  flight  rules  (VFR). 

Weather  for  the  mission  is  winds  calm,  1500  feet  overcast,  3  statute  miles  visibility  (ETA 
through  1  hour). 

Use  ATC  frequencies  per  the  DOD  FLIP.  FARP  1  frequency  is  34.15. 

Special  instructions  to  simulator  operator:  None 
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Table  B-l. 

Scenario  A,  nominal  condition  run  waypoint  card. 


WPT 

WPT  Description/ 
Coordinates 

Course 

ETE 

Maneuver  Description 

Distance 

(nm/km) 

!■ 

VMC  takeoff  230°  from 
Campbell  Army  Airfield 

i 

Road  Intersection 
N36°32.91’ 
W087°39.93’ 

226° 

Cruise  Flight 

900’  MSL,  100  KIAS 

10.5/19.3 

2 

Mouth  of  Inlet 
(Bridge) 
N36°28.52’ 
W087°44. 1 1  ’ 

239° 

Cruise  Flight 

1300’  MSL,  110  KIAS 

7.1/13.2 

3 

FARP  1 
N36°25.00’ 
W087°49.99’ 

240° 

Cruise  Flight 

1100’  MSL,  90  KIAS 

5.8/10.8 
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Scenario  B:  nominal  condition  run 


The  simulator  operator  will: 

1.  Set  visibility  to  3  statute  miles,  ceilings  to  1500  feet,  winds  and  turbulence  to  zero. 

2.  Set  time  of  day  at  12  noon. 

3.  Disable  the  GPS. 

Pre-mission  briefing  (read  to  participating  pilot): 

You  are  assigned  the  mission  of  transporting  several  boxes  containing  aircraft  parts  from  FARP  1 
to  FARP  2.  Your  aircraft  has  a  takeoff  weight  of  18,500  pounds.  Due  to  security  concerns,  a 
special  VMC  flight  corridor  at  terrain  flight  altitudes  has  been  established.  Your  flight  route  will 
over-fly  suspected  unmarked  mine  fields,  therefore,  you  should  not  decide  to  land  the  aircraft 
along  the  route  unless  there  are  no  other  alternatives.  The  GPS  is  inoperative  and  you  must 
navigate  by  pilotage  and  dead  reckoning.  You  must  follow  altitude,  airspeed  and  ground  track 
restrictions  in  accordance  with  the  waypoint  card  in  order  to  comply  with  the  corridor 
requirements.  Maintain  visual  flight  rules  (VFR). 

Weather  for  the  mission  is  winds  calm,  1500  feet  overcast,  3  statute  miles  visibility  (ETA 
through  1  hour). 

FARP  1  frequency  is  34.15  and  FARP  2  is  43.35.  Use  ATC  frequencies  per  the  DOD  FLIP  if 
necessary. 

Special  instructions  to  simulator  operator:  Approximately  2  minutes  into  the  flight,  activate  a 
Decreasing  %  RPM  R.  This  will  require  ECU  lockout  operations. 


132 


Table  B-2. 

Scenario  B,  nominal  condition  run  waypoint  card. 


WPT 

WPT 

Description/ 

Coordinates 

Course 

ETE 

Maneuver  Description 

Distance 

(nm/km) 

■ 

VMC  takeoff  from 
FARP  1 

4 

Road 

Intersection 

N36°28.77’ 

W087°53.31’ 

332° 

Cruise  Flight 

1000’  MSL,  80  KIAS 

4. 6/8. 6 

5 

End  of  Inlet 
N36°34.75’ 
W088°00.67’ 

322° 

Cruise  Flight 

1400’  MSL,  90  KIAS 

8.3/15.5 

6 

FARP2 

N36°40.27’ 

W088°02.98’ 

348° 

Cruise  Flight 

1100’  MSL,  80  KIAS 

5.7/10.7 
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Scenario  C:  off-nominal  condition  run 


The  simulator  operator  will: 

1.  Set  visibility  to  1  statute  mile,  ceilings  to  250  feet,  winds  to  zero  and  turbulence  to  Level  1. 

2.  Set  time  of  day  at  12  noon. 

Pre-mission  briefing  (read  to  participating  pilot): 

You  are  assigned  the  mission  of  repositioning  the  aircraft  from  FARP  2  to  Campbell  Army 
Airfield  for  AVIM-level  maintenance.  Due  to  the  weather,  you  have  been  briefed  to  proceed  via  a 
preplanned  route  that,  due  to  security  concerns,  must  be  followed.  A  special  VMC  flight  corridor, 
at  contour  flight  altitudes,  has  been  established  to  Campbell  Army  Airfield.  You  must  follow 
altitude,  airspeed  and  ground  track  restrictions  in  accordance  with  the  waypoint  card  in  order  to 
comply  with  the  corridor  requirements. 

Weather  for  the  mission  is  winds  calm,  250  feet  overcast,  1  statute  mile  visibility,  with  light 
turbulence  (ETA  through  1  hour).  Weather  at  Seaside  Army  Airfield,  which  can  be  used  as  an 
alternate  if  necessary,  is  forecast  to  be  winds  calm,  1500  feet  overcast,  3  statute  miles  visibility 
(ETA  through  1  hour). 

FARP  2  frequency  is  43.35.  Use  ATC  frequencies  per  the  DOD  FLIP. 

Special  instructions  to  simulator  operator:  Approximately  2  minutes  into  the  flight,  reduce 
visibility  to  0.50  sm.  Within  the  next  minute,  before  a  decision  can  be  made  whether  to  abort  or 
land,  reduce  visibility  to  zero  causing  inadvertent  entry  into  LMC.  When  asked,  provide  the 
aircraft  with  an  IFR  clearance  via  radar  vectors,  and  advise  the  crew  to  expect  the  ILS  Runway 
23  Approach  into  the  airfield.  See  Simulator  Operator  waypoint  card  for  altitude,  airspeed,  and 
heading  guidance.  Approximately  1  minute  into  task  number  five,  activate  the  number  one 
engine  light  with  no  other  indication  of  engine  failure,  do  not  fail  the  engine. 
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Give  this  card  to  flight  crew: 


Table  B-3. 


Scenario  C,  off-nominal  condition  run  waypoint  card. 


WPT 

WPT  Description/ 
Coordinates 

Course 

ETE 

Maneuver 

Description 

Distance 

(nm/km) 

VMC  takeoff  from 
FARP2 

7 

Mouth  of  Inlet 
N36°41.56’ 
W087°56.04’ 

085° 

Cruise  Flight 

1000’  MSL,  120 
KIAS 

5.7/10.7 

8 

Road  Bend 
N36°43.43’ 
W087°43.25’ 

086° 

Cruise  Flight 

1500’  MSL,  110 
KIAS 

10.5/19.2 

9 

Campbell  Army 
Airfield 
N36°40.29’ 
W087°29.29’ 

112° 

Cruise  Flight 

1100’  MSL,  120 
KIAS 

11.6/21.5 
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For  use  by  simulator  operator: 


Table  B-4. 

Simulator  operator  route  card. 


Task 

Maneuver 

Maneuver  Description 

~ETE 

1 

Initial  Climb  & 
Acceleration 
Instructions 

Turn  to  heading  070  and  climb 
straight  ahead  to  2000’  MSL,  at 
120  at  1000  fpm. 

1+30 

2 

Straight  and  Level  Flight 

Maintain  heading  070,  2000’ 
MSL,  120  KIAS. 

1+00 

3 

Straight  Climb 

Climb  straight  ahead  on  heading 
070,  from  2000’  to  4000’  MSL  at 
1000  fpm,  maintain  120  KIAS. 

2+00 

4 

Straight  and  Level  Flight 

Maintain  heading  070, 4000’ 
MSL,  120  KIAS. 

2+00 

5 

Straight  Descent 

Descend  straight  ahead  on  heading 
070,  from  4000’  to  2100’  MSL  at 
500  fpm,  maintain  120  KIAS. 

4+00 

6 

Straight  and  Level  Flight 

Maintain  heading  070,  2100’ 
MSL,  120  KIAS. 

1+00 

7 

Right  Standard  Rate 
Turn 

Turn  right  from  heading  070  to 
heading  140,  maintain  2100’  MSL 
and  120  KIAS  and  intercept  the 
ILS  23  inbound  course.  Cleared  for 
the  approach. 

0+30 

8 

ILS  23,  Campbell  AAF 

Maintain  inbound  course  225°, 
glideslope  and  120  KIAS. 

2+24 

136 


Campbell  Army  Airfield  approaches 


* 


* 
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Table  B-5. 

Campbell  Army  Airfield  approach  minimums. 


RADAR  INSTRUMENT  APPROACH  MINIMUMS 


CAMPBELL  AAF  (KHOP) 

RADMINS 

CAMPBELL  AAF  (KHOP),  (FORT  CAM  PBE  LL),  KY  (03079  USA)  ELEV  573 

RADAR  O -(E)  134.350*  237.5x  395.9x  258.3x  290.9x 


DH/ 

HAT/ 

RWY 

GS/TCH/RPI 

CAT 

M  DA-VIS 

HAA 

CEIL-VIS 

PAR 

23 

3.0/62/1183 

AB 

773/24 

200 

(200-%) 

CDE 

773/40 

200 

(200-%) 

ASR 

23 

AB 

980/40 

407 

(500-%) 

CD 

980/50 

407 

(500-1) 

E 

980/60 

407 

(500-1%) 

5® 

ABC 

940-1 

383 

(400-1) 

DE 

940-1% 

383 

(400-1%) 

CIR  & 

5,  23 

A 

1020-1 

447 

(500-1) 

B 

1040-1 

467 

(500-1) 

C 

1040-1% 

467 

(500-1%) 

DE 

1140-2 

567 

(600-2) 

®  Civil  GCA  132.025.  ®  Apch  not  auth  when  R-3701,  R-3702A,  in  use.  'I'  Cir  not  auth  SE  of  Rwy  5-23. 


RADAR  INSTRUMENT  APPROACH  MINIMUMS 


1 
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FORT  CAMPBELL,  KENTUCKY 


IOCI-FKP  I  APCHCRS 

111.8 


CAT  E  procedure  turn  not  authorized.  Radar  required. 
♦Circling  not  authorized  SE  Rwy  5-23. 


AUS 

125.175  308.4 


ILS  RWY  23 


AL-679  [USA]  CAMPBELL  AAF  (KHOP) 


““  I  MISSED  APPROACH:  Climb  to  1 200  then  dimbing  right 
torn  to  2500  direct  to  FK  LOM  and  hold. 


ASR/PAR 


CAMPBELL  APP  CON 

118.1  260.525 

CAMPBEU  TOWER 

120.9  27S.B 

GNDCON 

121.8  288.8 

CLNC  DEL 

138.8 

_ - 

— -  — • 

o 

CENTRAL  CmY  . 

109.8  CCT  =:=!  / 

Chan  35  „  7 


- IAF/LOM - 

AIRBE 

273  FK  HT-JL 


HEATE 

A~.?S°0 

/l03*- 

(*-3) 


LOCALIZER  1 1 1.9 
l-FKP  _1 


\  R-3702  A-B  ^g,rfrm< 

NOTE:  Additional  restrictions  ore 
in  affect  when  restricted 
areas  are  aclhre. 

ADF  or  RADAR  REQUIRED 


ELEV  573 


r 

>49 

Ur-*”5' 

1  n  ?inn 


Remain 
within  10  NM 


225*4.4  NM 

from  FA^ 

COPTER 

TD7E  / 

ALIGHTING 

573  &A 

Vj^| |  ^  2100 


rU^gnscai 


CATEGORY 


S-  ILS  23 


S-LOC  23 


C  D 


773/24  200  (200-14)  |  773/40  200  (200-*) 


960/40  387  (400  *1 


CIRCLING*  1040-1  447(500-1) 


S-PAR  23 


773/24  200  (200- >4) 


FORT  CAMPBELL,  KENTUCKY 
Arndt  1 0  04104 


Rwy  5  Ids  10,905 
960/50  3B7  (400-1)  HIRL  Rwy  5-23 


FAFtoMAPiUNM 


1140-2  547  1400-2) 


773/40  200  (200-*)  GS3. 


Y  34®40,N-B7*30’W  CAMPBELL  AAF  (KHOP) 

ILS  RWY  23 

Figure  B-2.  Campbell  Army  Airfield  ILS  RWY  23. 


IEHIEE3EB3E3IE3 
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Plan  view  of  flight  legs 


(Nl  to  Scale) 

FARP  13  =  Forward  Area  Refueling  Point 


Figure  B-3.  Plan  view  of  flight  legs. 


t 
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Helicopter  participant  effects  [p-  Values) 


Table  B-6. 

Helicopter  participant  effects  on  communications  (nominal). 

* 


|  Nominal 

Fit  Hrs 

Familiarity 

Hrs 

Spread 

UH-60 

Qual 

|rgniTTrn 

MSM 

Order 

(/> 

•*-» 

Relevant  Info 

0.43 

0.15 

0.48 

0.01 

0.28 

0.78 

0.76 

CO 

o 

Non  Relevant  Info 

XT 

0 

QC 

Total  Info 

0.43 

0.15 

0.48 

0.01 

0.28 

0.78 

0.76 

Action 

0.78 

0.66 

0.43 

0.19 

0.29 

0.88 

0.85 

(0 

i_ 

Relevant  Info 

0.31 

0.25 

0.49 

0.36 

0.92 

0.69 

0.26 

C) 

w 

Non  Relevant  Info 

0.46 

0.89 

0.08 

0.15 

0.90 

0.90 

0.39 

c 

CD 

Action 

0.07 

0.60 

0.05 

0.05 

0.38 

0.74 

0.54 

i- 

Total 

0.49 

0.41 

0.98 

0.77 

0.91 

0.73 

0.31 

c 

0) 

E 

General 

0.00 

0.27 

0.00 

0.73 

0.71 

0.72 

0.22 

O 

O) 

T3 

0> 

5 

Specific 

0.51 

0.81 

0.11 

0.55 

0.85 

0.84 

0.98 

c 

o 

< 

Total 

0.01 

0.56 

0.00 

0.88 

0.90 

0.92 

0.47 

Total 

Comms 

0.69 

0.95 

0.27 

0.68 

0.83  [ 

0.89 

0.83 

Anticipation 

Ratio 

0.37 

0.68 

0.76 

0.50 

0.88 

0.45 

0.15  | 

Table  B-7. 

Helicopter  participant  effects  on  communications  (off-nominal). 


|  Off  Nominal 

Fit  Hrs 

Familiarity 

Hrs 

Spread 

UH-60 

Qual 

Hrs  with 
Partner 

w 

Relevant  Info 

0.91 

0.05 

0.56 

0.24 

0.91 

0> 

Non  Relevant  Info 

0.90 

0.45 

0.35 

0.54 

c 

0.80 

0.09 

0.24 

0.50 

0.61 

CC 

Action 

0.18 

0.60 

0.07 

0.11 

0.98 

Relevant  Info 

0.17 

0.62 

0.18 

0.79 

0.23 

W 

h. 

0) 

M— 

CO 

c 

2 

Non  Relevant  Info 

0.46 

0.91 

0.73 

0.77 

0.93 

Action 

0.89 

0.72 

0.88 

0.11 

0.47 

1- 

Total 

0.18 

0.53 

0.16 

0.58 

0.27 

0) 

5 

o 

General 

0.53 

0.55 

0.36 

0.98 

0.94 

£ 

Specific 

0.75 

0.82 

0.27 

0.25 

0.65 

< 

Total 

0.78 

0.44 

0.93 

0.52 

0.74 

Total 

Comms 

0.10 

0.42 

0.10 

0.28 

0.53 

Anticipation 

Ratio 

0.05 

0.61 

0.04 

0.08 

0.37 
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Table  B-8. 

Helicopter  participant  effects  on  performance. 


Training 

Flight  Hours 

Flight  Hours  Spread 

UH-60  Qualifications 

Familiarity 

Hrs.  with  Crew 

Order 


Diagnosis 

Time 

Radio  Calls 

Nav  Errors 

ToTa T 
Errors 

X 

0.04 

0.47 

0.54 

0.63 

0.09 

0.38 

0.45 

0.28 

0.43 

0.22 

0.31 

0.27 

0.14 

0.14 

0.36 

0.55 

0.47 

0.54 

0.63 

0.79 

0.80 

0.29 

0.89 

0.58 

0.93 

0.69 

0.08 

0.95 

0.13 

X 

0.14 

0.81 

1.00 

0.81 

